[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:54:34 dist-test-slave-8nrt krb5kdc[11202](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:54:34 dist-test-slave-8nrt krb5kdc[11202](info): set up 2 sockets
Sep 29 10:54:34 dist-test-slave-8nrt krb5kdc[11202](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 10:54:35.552912 11195 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Sep 29 10:54:35 dist-test-slave-8nrt krb5kdc[11202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:54:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:54:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
I20260929 10:54:35.670361 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:36529
--webserver_interface=127.10.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:46739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:36529
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:54:35.752018 11215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:54:35.752241 11215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:54:35.752265 11215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:54:35.753365 11215 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:54:35.753407 11215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:54:35.753423 11215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:54:35.753438 11215 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:54:35.753453 11215 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:54:35.754859 11215 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:46739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:36529
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:36529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:54:35.755070 11215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:54:35.755252 11215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:54:35.760247 11223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:35.760305 11221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:35.760283 11220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:35.760489 11215 server_base.cc:1034] running on GCE node
I20260929 10:54:35.760759 11215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:54:35.761026 11215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:54:35.762173 11215 hybrid_clock.cc:648] HybridClock initialized: now 1790679275762157 us; error 30 us; skew 500 ppm
Sep 29 10:54:35 dist-test-slave-8nrt krb5kdc[11202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679275, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:54:35.764566 11215 init.cc:377] Logged in from keytab as kudu/127.10.238.254@KRBTEST.COM (short username kudu)
I20260929 10:54:35.765050 11215 webserver.cc:460] Webserver started at http://127.10.238.254:38155/ using document root <none> and password file <none>
I20260929 10:54:35.765353 11215 fs_manager.cc:359] Metadata directory not provided
I20260929 10:54:35.765424 11215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:54:35.765592 11215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:54:35.766640 11215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/master-0/data/instance:
uuid: "2bf83daa8dd0434a9c19b75e473f0e75"
format_stamp: "Formatted at 2026-09-29 10:54:35 on dist-test-slave-8nrt"
server_key: "c12a7bedc43de07ab7e66b220837d947"
server_key_iv: "93a85ad72a583fd2939b20a18e67e048"
server_key_version: "encryptionkey@0"
I20260929 10:54:35.767037 11215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/master-0/wal/instance:
uuid: "2bf83daa8dd0434a9c19b75e473f0e75"
format_stamp: "Formatted at 2026-09-29 10:54:35 on dist-test-slave-8nrt"
server_key: "c12a7bedc43de07ab7e66b220837d947"
server_key_iv: "93a85ad72a583fd2939b20a18e67e048"
server_key_version: "encryptionkey@0"
I20260929 10:54:35.768432 11215 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 10:54:35.769200 11230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:35.769475 11215 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:54:35.769579 11215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "2bf83daa8dd0434a9c19b75e473f0e75"
format_stamp: "Formatted at 2026-09-29 10:54:35 on dist-test-slave-8nrt"
server_key: "c12a7bedc43de07ab7e66b220837d947"
server_key_iv: "93a85ad72a583fd2939b20a18e67e048"
server_key_version: "encryptionkey@0"
I20260929 10:54:35.769659 11215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:54:35.777776 11215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:54:35.779479 11215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:54:35.779626 11215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:54:35.783818 11215 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.254:36529
I20260929 10:54:35.783846 11282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.254:36529 every 8 connection(s)
I20260929 10:54:35.784170 11215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:54:35.785000 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 11215
I20260929 10:54:35.785107 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/master-0/wal/instance
I20260929 10:54:35.785146 11283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:54:35.785897 11195 external_mini_cluster.cc:1428] Setting key eb0051c7ee17ca509dcc4108221df36d
I20260929 10:54:35.788004 11283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bf83daa8dd0434a9c19b75e473f0e75: Bootstrap starting.
I20260929 10:54:35.788568 11283 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2bf83daa8dd0434a9c19b75e473f0e75: Neither blocks nor log segments found. Creating new log.
I20260929 10:54:35.788913 11283 log.cc:824] T 00000000000000000000000000000000 P 2bf83daa8dd0434a9c19b75e473f0e75: Log is configured to *not* fsync() on all Append() calls
I20260929 10:54:35.789700 11283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bf83daa8dd0434a9c19b75e473f0e75: No bootstrap required, opened a new log
I20260929 10:54:35.791606 11283 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2bf83daa8dd0434a9c19b75e473f0e75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bf83daa8dd0434a9c19b75e473f0e75" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 36529 } }
I20260929 10:54:35.791772 11283 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2bf83daa8dd0434a9c19b75e473f0e75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:54:35.791806 11283 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2bf83daa8dd0434a9c19b75e473f0e75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bf83daa8dd0434a9c19b75e473f0e75, State: Initialized, Role: FOLLOWER
I20260929 10:54:35.791917 11283 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bf83daa8dd0434a9c19b75e473f0e75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bf83daa8dd0434a9c19b75e473f0e75" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 36529 } }
I20260929 10:54:35.791983 11283 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2bf83daa8dd0434a9c19b75e473f0e75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:54:35.792011 11283 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2bf83daa8dd0434a9c19b75e473f0e75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:54:35.792053 11283 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2bf83daa8dd0434a9c19b75e473f0e75 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:54:35.792645 11283 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2bf83daa8dd0434a9c19b75e473f0e75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bf83daa8dd0434a9c19b75e473f0e75" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 36529 } }
I20260929 10:54:35.792742 11283 leader_election.cc:302] T 00000000000000000000000000000000 P 2bf83daa8dd0434a9c19b75e473f0e75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bf83daa8dd0434a9c19b75e473f0e75; no voters:
I20260929 10:54:35.792912 11283 leader_election.cc:288] T 00000000000000000000000000000000 P 2bf83daa8dd0434a9c19b75e473f0e75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:54:35.792933 11288 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2bf83daa8dd0434a9c19b75e473f0e75 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:54:35.793043 11288 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2bf83daa8dd0434a9c19b75e473f0e75 [term 1 LEADER]: Becoming Leader. State: Replica: 2bf83daa8dd0434a9c19b75e473f0e75, State: Running, Role: LEADER
I20260929 10:54:35.793123 11288 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bf83daa8dd0434a9c19b75e473f0e75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bf83daa8dd0434a9c19b75e473f0e75" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 36529 } }
I20260929 10:54:35.793156 11283 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bf83daa8dd0434a9c19b75e473f0e75 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:54:35.793457 11288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bf83daa8dd0434a9c19b75e473f0e75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bf83daa8dd0434a9c19b75e473f0e75. Latest consensus state: current_term: 1 leader_uuid: "2bf83daa8dd0434a9c19b75e473f0e75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bf83daa8dd0434a9c19b75e473f0e75" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 36529 } } }
I20260929 10:54:35.793537 11288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bf83daa8dd0434a9c19b75e473f0e75 [sys.catalog]: This master's current role is: LEADER
I20260929 10:54:35.793468 11290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bf83daa8dd0434a9c19b75e473f0e75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2bf83daa8dd0434a9c19b75e473f0e75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bf83daa8dd0434a9c19b75e473f0e75" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 36529 } } }
I20260929 10:54:35.793694 11290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bf83daa8dd0434a9c19b75e473f0e75 [sys.catalog]: This master's current role is: LEADER
I20260929 10:54:35.793758 11294 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:54:35.794283 11294 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:54:35.796087 11294 catalog_manager.cc:1347] Generated new cluster ID: 66301fe811b141cab68fce151bcbe9a7
I20260929 10:54:35.796136 11294 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 10:54:35 dist-test-slave-8nrt krb5kdc[11202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:54:35.803117 11285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:35.786833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51538 (local address 127.10.238.254:36529)
0929 10:54:35.787012 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:35.787021 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:35.787239 (+ 218us) server_negotiation.cc:407] Connection header received
0929 10:54:35.789491 (+ 2252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:35.789499 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:35.789556 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:35.789694 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:35.791110 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:35.792497 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:35.793889 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:35.794422 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:35.794550 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:35.798940 (+ 4390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:35.798944 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:35.798945 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:35.798956 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:35.800901 (+ 1945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:35.801429 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:35.801433 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:35.801434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:35.801554 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:35.801857 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:35.801858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:35.801859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:35.802160 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:35.802320 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:35.802616 (+ 296us) server_negotiation.cc:299] Negotiation successful
0929 10:54:35.802916 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":81,"threads_started":1}
I20260929 10:54:35.812873 11294 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:54:35.813602 11294 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:54:35.823010 11294 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2bf83daa8dd0434a9c19b75e473f0e75: Generated new TSK 0
I20260929 10:54:35.823165 11294 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.238.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab.
Entry for principal kudu/127.10.238.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab.
I20260929 10:54:35.858132 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.193:0
--local_ip_for_outbound_sockets=127.10.238.193
--webserver_interface=127.10.238.193
--webserver_port=0
--tserver_master_addrs=127.10.238.254:36529
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab
--principal=kudu/127.10.238.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:46739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:54:35.957113 11309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:54:35.957324 11309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:54:35.957348 11309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:54:35.958431 11309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:54:35.958464 11309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:54:35.958490 11309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.193
I20260929 10:54:35.959847 11309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:46739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab
--principal=kudu/127.10.238.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.238.193
--webserver_port=0
--tserver_master_addrs=127.10.238.254:36529
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.193
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:54:35.960026 11309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:54:35.960208 11309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:54:35.965181 11317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:35.965191 11314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:35.965191 11315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:35.965271 11309 server_base.cc:1034] running on GCE node
I20260929 10:54:35.965724 11309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:54:35.965956 11309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:54:35.967151 11309 hybrid_clock.cc:648] HybridClock initialized: now 1790679275967124 us; error 41 us; skew 500 ppm
Sep 29 10:54:35 dist-test-slave-8nrt krb5kdc[11202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679275, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:54:35.969758 11309 init.cc:377] Logged in from keytab as kudu/127.10.238.193@KRBTEST.COM (short username kudu)
I20260929 10:54:35.970192 11309 webserver.cc:460] Webserver started at http://127.10.238.193:41841/ using document root <none> and password file <none>
I20260929 10:54:35.970435 11309 fs_manager.cc:359] Metadata directory not provided
I20260929 10:54:35.970479 11309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:54:35.970638 11309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:54:35.971702 11309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/ts-0/data/instance:
uuid: "d74b7309ac1f4c48b602f125ddeff4c3"
format_stamp: "Formatted at 2026-09-29 10:54:35 on dist-test-slave-8nrt"
server_key: "d82d09c5c11d2aef6291efe3cb8bdeb5"
server_key_iv: "b08b963785a90f28409e87aaa4ddc558"
server_key_version: "encryptionkey@0"
I20260929 10:54:35.972098 11309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/ts-0/wal/instance:
uuid: "d74b7309ac1f4c48b602f125ddeff4c3"
format_stamp: "Formatted at 2026-09-29 10:54:35 on dist-test-slave-8nrt"
server_key: "d82d09c5c11d2aef6291efe3cb8bdeb5"
server_key_iv: "b08b963785a90f28409e87aaa4ddc558"
server_key_version: "encryptionkey@0"
I20260929 10:54:35.973448 11309 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 10:54:35.974107 11324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:35.974267 11309 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:54:35.974371 11309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/ts-0/wal
uuid: "d74b7309ac1f4c48b602f125ddeff4c3"
format_stamp: "Formatted at 2026-09-29 10:54:35 on dist-test-slave-8nrt"
server_key: "d82d09c5c11d2aef6291efe3cb8bdeb5"
server_key_iv: "b08b963785a90f28409e87aaa4ddc558"
server_key_version: "encryptionkey@0"
I20260929 10:54:35.974447 11309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:54:35.993574 11309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:54:35.996001 11309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:54:35.996162 11309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:54:35.996423 11309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:54:35.996734 11309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:54:35.996786 11309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:35.996826 11309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:54:35.996853 11309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:36.002749 11309 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.193:43277
I20260929 10:54:36.002799 11437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.193:43277 every 8 connection(s)
I20260929 10:54:36.003093 11309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
I20260929 10:54:36.003639 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 11309
I20260929 10:54:36.003741 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/ts-0/wal/instance
I20260929 10:54:36.003928 11195 external_mini_cluster.cc:1428] Setting key f20723efeb3700c548bbc5c9e1a1f49f
Sep 29 10:54:36 dist-test-slave-8nrt krb5kdc[11202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679275, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.193@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.238.194@KRBTEST.COM; defaulting to no policy
I20260929 10:54:36.013129 11285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:36.003800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:37209 (local address 127.10.238.254:36529)
0929 10:54:36.003848 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:36.003850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:36.004134 (+ 284us) server_negotiation.cc:407] Connection header received
0929 10:54:36.004738 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:36.004739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:36.004751 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:36.004819 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:36.005950 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:36.006451 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:36.007524 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:36.007719 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:36.007772 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:36.010265 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:36.010266 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:36.010267 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:36.010274 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:36.011702 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:36.012099 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:36.012100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:36.012100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:36.012164 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:36.012447 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:36.012448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:36.012448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:36.012562 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:36.012636 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:36.012949 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 10:54:36.013037 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 10:54:36.013324 11440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:36.003799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:36529 (local address 127.10.238.193:37209)
0929 10:54:36.003995 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:36.004047 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:36.004630 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:36.004873 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:36.004876 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:36.005562 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:54:36.005821 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:36.005829 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:36.006540 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:36.006541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:36.007449 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:36.007452 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:36.007803 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:36.007804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:36.007955 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:36.008439 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:54:36.008449 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:54:36.010155 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:54:36.011790 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:36.011798 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:36.011800 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:36.012021 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:36.012255 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:36.012256 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:36.012257 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:36.012366 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:36.012630 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:54:36.012636 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:54:36.012776 (+ 140us) client_negotiation.cc:769] Sending connection context
0929 10:54:36.012928 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 10:54:36.013081 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":97,"threads_started":1}
I20260929 10:54:36.013911 11438 heartbeater.cc:347] Connected to a master server at 127.10.238.254:36529
I20260929 10:54:36.014014 11438 heartbeater.cc:464] Registering TS with master...
I20260929 10:54:36.014197 11438 heartbeater.cc:511] Master 127.10.238.254:36529 requested a full tablet report, sending...
I20260929 10:54:36.014709 11247 ts_manager.cc:194] Registered new tserver with Master: d74b7309ac1f4c48b602f125ddeff4c3 (127.10.238.193:43277)
I20260929 10:54:36.015319 11247 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.193@KRBTEST.COM'} at 127.10.238.193:37209
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab.
Entry for principal kudu/127.10.238.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab.
I20260929 10:54:36.029762 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.194:0
--local_ip_for_outbound_sockets=127.10.238.194
--webserver_interface=127.10.238.194
--webserver_port=0
--tserver_master_addrs=127.10.238.254:36529
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab
--principal=kudu/127.10.238.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:46739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:54:36.112543 11443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:54:36.112782 11443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:54:36.112808 11443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:54:36.114094 11443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:54:36.114140 11443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:54:36.114169 11443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.194
I20260929 10:54:36.115780 11443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:46739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab
--principal=kudu/127.10.238.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.238.194
--webserver_port=0
--tserver_master_addrs=127.10.238.254:36529
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.194
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:54:36.115967 11443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:54:36.116135 11443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:54:36.121070 11448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:36.121138 11451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:36.121134 11449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:36.121297 11443 server_base.cc:1034] running on GCE node
I20260929 10:54:36.121577 11443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:54:36.121810 11443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:54:36.122969 11443 hybrid_clock.cc:648] HybridClock initialized: now 1790679276122949 us; error 55 us; skew 500 ppm
Sep 29 10:54:36 dist-test-slave-8nrt krb5kdc[11202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679276, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:54:36.125541 11443 init.cc:377] Logged in from keytab as kudu/127.10.238.194@KRBTEST.COM (short username kudu)
I20260929 10:54:36.126004 11443 webserver.cc:460] Webserver started at http://127.10.238.194:41055/ using document root <none> and password file <none>
I20260929 10:54:36.126228 11443 fs_manager.cc:359] Metadata directory not provided
I20260929 10:54:36.126293 11443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:54:36.126423 11443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:54:36.127485 11443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/ts-1/data/instance:
uuid: "e01d7315a2914964ae6c684ac4678a3a"
format_stamp: "Formatted at 2026-09-29 10:54:36 on dist-test-slave-8nrt"
server_key: "04e95d53de5380ce2c872a4ef3f5cfe0"
server_key_iv: "42537e97baf3dcb3c29c534aa769996b"
server_key_version: "encryptionkey@0"
I20260929 10:54:36.127852 11443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/ts-1/wal/instance:
uuid: "e01d7315a2914964ae6c684ac4678a3a"
format_stamp: "Formatted at 2026-09-29 10:54:36 on dist-test-slave-8nrt"
server_key: "04e95d53de5380ce2c872a4ef3f5cfe0"
server_key_iv: "42537e97baf3dcb3c29c534aa769996b"
server_key_version: "encryptionkey@0"
I20260929 10:54:36.129194 11443 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260929 10:54:36.129984 11458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:36.130185 11443 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 10:54:36.130326 11443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/ts-1/wal
uuid: "e01d7315a2914964ae6c684ac4678a3a"
format_stamp: "Formatted at 2026-09-29 10:54:36 on dist-test-slave-8nrt"
server_key: "04e95d53de5380ce2c872a4ef3f5cfe0"
server_key_iv: "42537e97baf3dcb3c29c534aa769996b"
server_key_version: "encryptionkey@0"
I20260929 10:54:36.130407 11443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:54:36.158643 11443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:54:36.161052 11443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:54:36.161213 11443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:54:36.161502 11443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:54:36.161813 11443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:54:36.161862 11443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:36.161903 11443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:54:36.161928 11443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:36.168076 11443 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.194:38759
I20260929 10:54:36.168120 11571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.194:38759 every 8 connection(s)
I20260929 10:54:36.168445 11443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
I20260929 10:54:36.174463 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 11443
I20260929 10:54:36.174583 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/ts-1/wal/instance
Sep 29 10:54:36 dist-test-slave-8nrt krb5kdc[11202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679276, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.194@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:54:36.174763 11195 external_mini_cluster.cc:1428] Setting key 2ec37779f479aae406ad0064d9dfe5ca
I20260929 10:54:36.178501 11285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:36.169183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:59105 (local address 127.10.238.254:36529)
0929 10:54:36.169291 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:36.169294 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:36.169500 (+ 206us) server_negotiation.cc:407] Connection header received
0929 10:54:36.169991 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:36.169993 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:36.170002 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:36.170055 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:36.170998 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:36.171495 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:36.172493 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:36.172740 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:36.172796 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:36.175301 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:36.175303 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:36.175304 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:36.175312 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:36.176827 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:36.177450 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:36.177452 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:36.177452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:36.177512 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:36.177793 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:36.177793 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:36.177794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:36.177912 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:36.177980 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:36.178328 (+ 348us) server_negotiation.cc:299] Negotiation successful
0929 10:54:36.178413 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 10:54:36.178694 11574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:36.169147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:36529 (local address 127.10.238.194:59105)
0929 10:54:36.169364 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:36.169403 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:36.169906 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:36.170109 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:36.170113 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:36.170709 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:54:36.170922 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:36.170927 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:36.171601 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:36.171602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:36.172421 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:36.172423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:36.172815 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:36.172816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:36.172911 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:36.173500 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:54:36.173509 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:54:36.175207 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:54:36.176955 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:36.176963 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:36.176965 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:36.177373 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:36.177588 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:36.177589 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:36.177590 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:36.177717 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:36.177984 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:54:36.177991 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:54:36.178169 (+ 178us) client_negotiation.cc:769] Sending connection context
0929 10:54:36.178281 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 10:54:36.178449 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":141,"threads_started":1}
I20260929 10:54:36.179134 11572 heartbeater.cc:347] Connected to a master server at 127.10.238.254:36529
I20260929 10:54:36.179231 11572 heartbeater.cc:464] Registering TS with master...
I20260929 10:54:36.179445 11572 heartbeater.cc:511] Master 127.10.238.254:36529 requested a full tablet report, sending...
I20260929 10:54:36.179840 11247 ts_manager.cc:194] Registered new tserver with Master: e01d7315a2914964ae6c684ac4678a3a (127.10.238.194:38759)
I20260929 10:54:36.180228 11247 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.194@KRBTEST.COM'} at 127.10.238.194:59105
WARNING: no policy specified for kudu/127.10.238.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab.
Entry for principal kudu/127.10.238.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab.
I20260929 10:54:36.199910 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.195:0
--local_ip_for_outbound_sockets=127.10.238.195
--webserver_interface=127.10.238.195
--webserver_port=0
--tserver_master_addrs=127.10.238.254:36529
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab
--principal=kudu/127.10.238.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:46739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:54:36.279850 11577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:54:36.280045 11577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:54:36.280067 11577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:54:36.281242 11577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:54:36.281340 11577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:54:36.281369 11577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.195
I20260929 10:54:36.282789 11577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:46739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab
--principal=kudu/127.10.238.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.238.195
--webserver_port=0
--tserver_master_addrs=127.10.238.254:36529
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.195
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:54:36.283025 11577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:54:36.283247 11577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:54:36.288162 11585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:36.288162 11583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:36.288144 11582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:36.288439 11577 server_base.cc:1034] running on GCE node
I20260929 10:54:36.288622 11577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:54:36.288811 11577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:54:36.289983 11577 hybrid_clock.cc:648] HybridClock initialized: now 1790679276289969 us; error 25 us; skew 500 ppm
Sep 29 10:54:36 dist-test-slave-8nrt krb5kdc[11202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679276, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:54:36.292285 11577 init.cc:377] Logged in from keytab as kudu/127.10.238.195@KRBTEST.COM (short username kudu)
I20260929 10:54:36.292811 11577 webserver.cc:460] Webserver started at http://127.10.238.195:40871/ using document root <none> and password file <none>
I20260929 10:54:36.293038 11577 fs_manager.cc:359] Metadata directory not provided
I20260929 10:54:36.293102 11577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:54:36.293236 11577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:54:36.294431 11577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/ts-2/data/instance:
uuid: "27e276dbf3264c5bb54233ffdb6b8151"
format_stamp: "Formatted at 2026-09-29 10:54:36 on dist-test-slave-8nrt"
server_key: "e981c37d1b477d3dc96a73dadabbf3e4"
server_key_iv: "82a3338fda6207ea2d7e3070687215ac"
server_key_version: "encryptionkey@0"
I20260929 10:54:36.294842 11577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/ts-2/wal/instance:
uuid: "27e276dbf3264c5bb54233ffdb6b8151"
format_stamp: "Formatted at 2026-09-29 10:54:36 on dist-test-slave-8nrt"
server_key: "e981c37d1b477d3dc96a73dadabbf3e4"
server_key_iv: "82a3338fda6207ea2d7e3070687215ac"
server_key_version: "encryptionkey@0"
I20260929 10:54:36.296085 11577 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 10:54:36.296761 11592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:36.296941 11577 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 10:54:36.297009 11577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/ts-2/wal
uuid: "27e276dbf3264c5bb54233ffdb6b8151"
format_stamp: "Formatted at 2026-09-29 10:54:36 on dist-test-slave-8nrt"
server_key: "e981c37d1b477d3dc96a73dadabbf3e4"
server_key_iv: "82a3338fda6207ea2d7e3070687215ac"
server_key_version: "encryptionkey@0"
I20260929 10:54:36.297058 11577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:54:36.308889 11577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:54:36.310755 11577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:54:36.310865 11577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:54:36.311035 11577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:54:36.311344 11577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:54:36.311378 11577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:36.311398 11577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:54:36.311445 11577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:36.317766 11577 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.195:35413
I20260929 10:54:36.317847 11705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.195:35413 every 8 connection(s)
I20260929 10:54:36.318143 11577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
Sep 29 10:54:36 dist-test-slave-8nrt krb5kdc[11202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679276, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.195@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:54:36.324314 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 11577
I20260929 10:54:36.324394 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790679274510869-11195-0/minicluster-data/ts-2/wal/instance
I20260929 10:54:36.324582 11195 external_mini_cluster.cc:1428] Setting key c3abe957316d5717e34059f0f091d9ce
I20260929 10:54:36.327280 11285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:36.318899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:40559 (local address 127.10.238.254:36529)
0929 10:54:36.318958 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:36.318960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:36.319182 (+ 222us) server_negotiation.cc:407] Connection header received
0929 10:54:36.319646 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:36.319647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:36.319657 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:36.319737 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:36.320623 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:36.321123 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:36.322148 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:36.322337 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:36.322399 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:36.324575 (+ 2176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:36.324577 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:36.324577 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:36.324583 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:36.326144 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:36.326504 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:36.326504 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:36.326505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:36.326555 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:36.326749 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:36.326749 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:36.326749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:36.326849 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:36.326901 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:36.327156 (+ 255us) server_negotiation.cc:299] Negotiation successful
0929 10:54:36.327218 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 10:54:36.327442 11708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:36.318882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:36529 (local address 127.10.238.195:40559)
0929 10:54:36.319055 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:36.319096 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:36.319567 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:36.319800 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:36.319804 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:36.320361 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:54:36.320551 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:36.320555 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:36.321202 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:36.321203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:36.322076 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:36.322078 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:36.322417 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:36.322418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:36.322516 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:36.322992 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:54:36.323001 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:54:36.324490 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:54:36.326210 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:36.326215 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:36.326217 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:36.326429 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:36.326596 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:36.326597 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:36.326597 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:36.326675 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:36.326915 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:54:36.326920 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:54:36.327044 (+ 124us) client_negotiation.cc:769] Sending connection context
0929 10:54:36.327133 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 10:54:36.327248 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":94,"threads_started":1}
I20260929 10:54:36.327879 11706 heartbeater.cc:347] Connected to a master server at 127.10.238.254:36529
I20260929 10:54:36.327992 11706 heartbeater.cc:464] Registering TS with master...
I20260929 10:54:36.328151 11706 heartbeater.cc:511] Master 127.10.238.254:36529 requested a full tablet report, sending...
I20260929 10:54:36.328469 11247 ts_manager.cc:194] Registered new tserver with Master: 27e276dbf3264c5bb54233ffdb6b8151 (127.10.238.195:35413)
I20260929 10:54:36.328799 11247 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.195@KRBTEST.COM'} at 127.10.238.195:40559
I20260929 10:54:36.336570 11195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 10:54:36 dist-test-slave-8nrt krb5kdc[11202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679276, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 10:54:36 dist-test-slave-8nrt krb5kdc[11202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679276, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.238.193@KRBTEST.COM
I20260929 10:54:36.359157 11713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:36.351680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46602 (local address 127.10.238.193:43277)
0929 10:54:36.351828 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:36.351834 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:36.351854 (+ 20us) server_negotiation.cc:407] Connection header received
0929 10:54:36.351881 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:36.351883 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:36.351897 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:36.351964 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:36.352560 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:36.353333 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:36.354054 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:36.354391 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:36.354446 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:36.356600 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:36.356608 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:36.356609 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:36.356615 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:36.358232 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:36.358544 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:36.358547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:36.358547 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:36.358601 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:36.358747 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:36.358748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:36.358748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:36.358867 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:36.358987 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:36.359020 (+ 33us) server_negotiation.cc:299] Negotiation successful
0929 10:54:36.359079 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":90,"threads_started":1}
W20260929 10:54:36.359547 11352 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46602
Sep 29 10:54:36 dist-test-slave-8nrt krb5kdc[11202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 10:54:36 dist-test-slave-8nrt krb5kdc[11202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.238.193@KRBTEST.COM
I20260929 10:54:36.380913 11713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:36.374422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46618 (local address 127.10.238.193:43277)
0929 10:54:36.374504 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:36.374506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:36.374559 (+ 53us) server_negotiation.cc:407] Connection header received
0929 10:54:36.374601 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:36.374602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:36.374611 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:36.374659 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:36.375180 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:36.375675 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:36.376370 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:36.376580 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:36.376781 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:36.378913 (+ 2132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:36.378914 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:36.378915 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:36.378921 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:36.380058 (+ 1137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:36.380347 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:36.380348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:36.380348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:36.380395 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:36.380549 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:36.380549 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:36.380549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:36.380631 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:36.380711 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:36.380789 (+ 78us) server_negotiation.cc:299] Negotiation successful
0929 10:54:36.380849 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 10:54:36.381464 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 11309
I20260929 10:54:36.386040 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 11443
I20260929 10:54:36.390427 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 11577
I20260929 10:54:36.396799 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 11215
2026-09-29T10:54:36Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1893 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:54:36 dist-test-slave-8nrt krb5kdc[11720](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:54:36 dist-test-slave-8nrt krb5kdc[11720](info): set up 2 sockets
Sep 29 10:54:36 dist-test-slave-8nrt krb5kdc[11720](info): commencing operation
krb5kdc: starting...
W20260929 10:54:37.428711 11195 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Sep 29 10:54:37 dist-test-slave-8nrt krb5kdc[11720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:54:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:54:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
I20260929 10:54:37.543742 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:44365
--webserver_interface=127.10.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:40199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:44365
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:54:37.624055 11733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:54:37.624310 11733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:54:37.624351 11733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:54:37.625667 11733 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:54:37.625746 11733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:54:37.625773 11733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:54:37.625799 11733 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:54:37.625833 11733 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:54:37.627328 11733 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:40199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:44365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:44365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:54:37.627619 11733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:54:37.627856 11733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:54:37.632810 11739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:37.632767 11741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:37.632967 11733 server_base.cc:1034] running on GCE node
W20260929 10:54:37.632784 11738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:37.633221 11733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:54:37.633567 11733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:54:37.634745 11733 hybrid_clock.cc:648] HybridClock initialized: now 1790679277634720 us; error 38 us; skew 500 ppm
Sep 29 10:54:37 dist-test-slave-8nrt krb5kdc[11720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679277, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:54:37.640136 11733 init.cc:377] Logged in from keytab as kudu/127.10.238.254@KRBTEST.COM (short username kudu)
I20260929 10:54:37.640605 11733 webserver.cc:460] Webserver started at http://127.10.238.254:44059/ using document root <none> and password file <none>
I20260929 10:54:37.640853 11733 fs_manager.cc:359] Metadata directory not provided
I20260929 10:54:37.640920 11733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:54:37.641069 11733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:54:37.642156 11733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/master-0/data/instance:
uuid: "cab67ce199414489b0f0561b0836b173"
format_stamp: "Formatted at 2026-09-29 10:54:37 on dist-test-slave-8nrt"
server_key: "accc4e5269fc0a12afff2cdbee93faff"
server_key_iv: "3608879a870460cf6c175eccdbfbb039"
server_key_version: "encryptionkey@0"
I20260929 10:54:37.642547 11733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/master-0/wal/instance:
uuid: "cab67ce199414489b0f0561b0836b173"
format_stamp: "Formatted at 2026-09-29 10:54:37 on dist-test-slave-8nrt"
server_key: "accc4e5269fc0a12afff2cdbee93faff"
server_key_iv: "3608879a870460cf6c175eccdbfbb039"
server_key_version: "encryptionkey@0"
I20260929 10:54:37.643860 11733 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 10:54:37.644569 11748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:37.644758 11733 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:54:37.644848 11733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "cab67ce199414489b0f0561b0836b173"
format_stamp: "Formatted at 2026-09-29 10:54:37 on dist-test-slave-8nrt"
server_key: "accc4e5269fc0a12afff2cdbee93faff"
server_key_iv: "3608879a870460cf6c175eccdbfbb039"
server_key_version: "encryptionkey@0"
I20260929 10:54:37.644933 11733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:54:37.658977 11733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:54:37.660727 11733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:54:37.660889 11733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:54:37.664958 11733 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.254:44365
I20260929 10:54:37.664989 11800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.254:44365 every 8 connection(s)
I20260929 10:54:37.665356 11733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:54:37.666188 11801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:54:37.668243 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 11733
I20260929 10:54:37.668336 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/master-0/wal/instance
I20260929 10:54:37.668511 11195 external_mini_cluster.cc:1428] Setting key 86e6647843d6203885d506f1c4b9d0d5
I20260929 10:54:37.668648 11801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cab67ce199414489b0f0561b0836b173: Bootstrap starting.
I20260929 10:54:37.669389 11801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cab67ce199414489b0f0561b0836b173: Neither blocks nor log segments found. Creating new log.
I20260929 10:54:37.669689 11801 log.cc:824] T 00000000000000000000000000000000 P cab67ce199414489b0f0561b0836b173: Log is configured to *not* fsync() on all Append() calls
I20260929 10:54:37.670380 11801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cab67ce199414489b0f0561b0836b173: No bootstrap required, opened a new log
I20260929 10:54:37.672103 11801 raft_consensus.cc:348] T 00000000000000000000000000000000 P cab67ce199414489b0f0561b0836b173 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab67ce199414489b0f0561b0836b173" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 44365 } }
I20260929 10:54:37.672256 11801 raft_consensus.cc:374] T 00000000000000000000000000000000 P cab67ce199414489b0f0561b0836b173 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:54:37.672314 11801 raft_consensus.cc:729] T 00000000000000000000000000000000 P cab67ce199414489b0f0561b0836b173 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab67ce199414489b0f0561b0836b173, State: Initialized, Role: FOLLOWER
I20260929 10:54:37.672461 11801 consensus_queue.cc:260] T 00000000000000000000000000000000 P cab67ce199414489b0f0561b0836b173 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab67ce199414489b0f0561b0836b173" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 44365 } }
I20260929 10:54:37.672542 11801 raft_consensus.cc:388] T 00000000000000000000000000000000 P cab67ce199414489b0f0561b0836b173 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:54:37.672569 11801 raft_consensus.cc:482] T 00000000000000000000000000000000 P cab67ce199414489b0f0561b0836b173 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:54:37.672603 11801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cab67ce199414489b0f0561b0836b173 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:54:37.673302 11801 raft_consensus.cc:504] T 00000000000000000000000000000000 P cab67ce199414489b0f0561b0836b173 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab67ce199414489b0f0561b0836b173" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 44365 } }
I20260929 10:54:37.673432 11801 leader_election.cc:302] T 00000000000000000000000000000000 P cab67ce199414489b0f0561b0836b173 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cab67ce199414489b0f0561b0836b173; no voters:
I20260929 10:54:37.673594 11801 leader_election.cc:288] T 00000000000000000000000000000000 P cab67ce199414489b0f0561b0836b173 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:54:37.673730 11806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cab67ce199414489b0f0561b0836b173 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:54:37.673857 11801 sys_catalog.cc:564] T 00000000000000000000000000000000 P cab67ce199414489b0f0561b0836b173 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:54:37.673902 11806 raft_consensus.cc:686] T 00000000000000000000000000000000 P cab67ce199414489b0f0561b0836b173 [term 1 LEADER]: Becoming Leader. State: Replica: cab67ce199414489b0f0561b0836b173, State: Running, Role: LEADER
I20260929 10:54:37.674008 11806 consensus_queue.cc:237] T 00000000000000000000000000000000 P cab67ce199414489b0f0561b0836b173 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab67ce199414489b0f0561b0836b173" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 44365 } }
I20260929 10:54:37.674391 11807 sys_catalog.cc:455] T 00000000000000000000000000000000 P cab67ce199414489b0f0561b0836b173 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cab67ce199414489b0f0561b0836b173" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab67ce199414489b0f0561b0836b173" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 44365 } } }
I20260929 10:54:37.674484 11807 sys_catalog.cc:458] T 00000000000000000000000000000000 P cab67ce199414489b0f0561b0836b173 [sys.catalog]: This master's current role is: LEADER
I20260929 10:54:37.674642 11808 sys_catalog.cc:455] T 00000000000000000000000000000000 P cab67ce199414489b0f0561b0836b173 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cab67ce199414489b0f0561b0836b173. Latest consensus state: current_term: 1 leader_uuid: "cab67ce199414489b0f0561b0836b173" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab67ce199414489b0f0561b0836b173" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 44365 } } }
I20260929 10:54:37.674696 11808 sys_catalog.cc:458] T 00000000000000000000000000000000 P cab67ce199414489b0f0561b0836b173 [sys.catalog]: This master's current role is: LEADER
I20260929 10:54:37.675000 11815 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 29 10:54:37 dist-test-slave-8nrt krb5kdc[11720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:54:37.675464 11815 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:54:37.677294 11815 catalog_manager.cc:1347] Generated new cluster ID: 9a7a6a79a0924b9c8d789fcf26174bfa
I20260929 10:54:37.677348 11815 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:54:37.679678 11804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:37.669077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42068 (local address 127.10.238.254:44365)
0929 10:54:37.669280 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:37.669289 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:37.669320 (+ 31us) server_negotiation.cc:407] Connection header received
0929 10:54:37.669852 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:37.669858 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:37.669908 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:37.670055 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:37.670732 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:37.671605 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:37.672396 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:37.673032 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:37.673141 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:37.676099 (+ 2958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:37.676104 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:37.676104 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:37.676116 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:37.678103 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:37.678499 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:37.678502 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:37.678502 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:37.678600 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:37.678820 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:37.678821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:37.678821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:37.679090 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:37.679241 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:37.679356 (+ 115us) server_negotiation.cc:299] Negotiation successful
0929 10:54:37.679481 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":83,"threads_started":1}
I20260929 10:54:37.692629 11815 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:54:37.693365 11815 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:54:37.701685 11815 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cab67ce199414489b0f0561b0836b173: Generated new TSK 0
I20260929 10:54:37.701858 11815 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.238.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab.
Entry for principal kudu/127.10.238.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab.
I20260929 10:54:37.732038 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.193:0
--local_ip_for_outbound_sockets=127.10.238.193
--webserver_interface=127.10.238.193
--webserver_port=0
--tserver_master_addrs=127.10.238.254:44365
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab
--principal=kudu/127.10.238.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:40199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:54:37.828125 11827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:54:37.828315 11827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:54:37.828338 11827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:54:37.829509 11827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:54:37.829550 11827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:54:37.829602 11827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.193
I20260929 10:54:37.831058 11827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:40199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab
--principal=kudu/127.10.238.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.238.193
--webserver_port=0
--tserver_master_addrs=127.10.238.254:44365
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.193
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:54:37.831293 11827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:54:37.831514 11827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:54:37.836448 11835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:37.836448 11832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:37.836531 11833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:37.836525 11827 server_base.cc:1034] running on GCE node
I20260929 10:54:37.836918 11827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:54:37.837186 11827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:54:37.838367 11827 hybrid_clock.cc:648] HybridClock initialized: now 1790679277838363 us; error 59 us; skew 500 ppm
Sep 29 10:54:37 dist-test-slave-8nrt krb5kdc[11720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679277, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:54:37.841003 11827 init.cc:377] Logged in from keytab as kudu/127.10.238.193@KRBTEST.COM (short username kudu)
I20260929 10:54:37.841513 11827 webserver.cc:460] Webserver started at http://127.10.238.193:38455/ using document root <none> and password file <none>
I20260929 10:54:37.841779 11827 fs_manager.cc:359] Metadata directory not provided
I20260929 10:54:37.841905 11827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:54:37.842046 11827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:54:37.843428 11827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/ts-0/data/instance:
uuid: "11dbb65fc43c4fa2b3e6a7d62131902d"
format_stamp: "Formatted at 2026-09-29 10:54:37 on dist-test-slave-8nrt"
server_key: "4a86b39c03f1b5fe24d6fb9bdd3d871c"
server_key_iv: "63e9ea41c71240ac9538cbbe8747431a"
server_key_version: "encryptionkey@0"
I20260929 10:54:37.843847 11827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/ts-0/wal/instance:
uuid: "11dbb65fc43c4fa2b3e6a7d62131902d"
format_stamp: "Formatted at 2026-09-29 10:54:37 on dist-test-slave-8nrt"
server_key: "4a86b39c03f1b5fe24d6fb9bdd3d871c"
server_key_iv: "63e9ea41c71240ac9538cbbe8747431a"
server_key_version: "encryptionkey@0"
I20260929 10:54:37.845369 11827 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 10:54:37.846216 11842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:37.846395 11827 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:54:37.846479 11827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/ts-0/wal
uuid: "11dbb65fc43c4fa2b3e6a7d62131902d"
format_stamp: "Formatted at 2026-09-29 10:54:37 on dist-test-slave-8nrt"
server_key: "4a86b39c03f1b5fe24d6fb9bdd3d871c"
server_key_iv: "63e9ea41c71240ac9538cbbe8747431a"
server_key_version: "encryptionkey@0"
I20260929 10:54:37.846546 11827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:54:37.860913 11827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:54:37.863106 11827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:54:37.863212 11827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:54:37.863417 11827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:54:37.863708 11827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:54:37.863760 11827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:37.863782 11827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:54:37.863816 11827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:37.869642 11827 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.193:39643
I20260929 10:54:37.869690 11955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.193:39643 every 8 connection(s)
I20260929 10:54:37.869982 11827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
Sep 29 10:54:37 dist-test-slave-8nrt krb5kdc[11720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679277, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.193@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:54:37.877074 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 11827
I20260929 10:54:37.877182 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/ts-0/wal/instance
I20260929 10:54:37.877385 11195 external_mini_cluster.cc:1428] Setting key 60ac99b629db9fd40efcd1b1f717ad36
I20260929 10:54:37.879240 11958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:37.870612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:44365 (local address 127.10.238.193:45321)
0929 10:54:37.870789 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:37.870826 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:37.871745 (+ 919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:37.871972 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:37.871975 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:37.872221 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:54:37.872397 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:37.872401 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:37.873038 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:37.873038 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:37.873978 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:37.873981 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:37.874356 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:37.874357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:37.874452 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:37.874987 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:54:37.874995 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:54:37.876330 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:54:37.877981 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:37.877988 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:37.877990 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:37.878225 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:37.878430 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:37.878430 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:37.878430 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:37.878535 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:37.878806 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:54:37.878810 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:54:37.878911 (+ 101us) client_negotiation.cc:769] Sending connection context
0929 10:54:37.878986 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 10:54:37.879091 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":86,"threads_started":1}
I20260929 10:54:37.879263 11804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:37.870639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:45321 (local address 127.10.238.254:44365)
0929 10:54:37.870702 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:37.870704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:37.870946 (+ 242us) server_negotiation.cc:407] Connection header received
0929 10:54:37.871832 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:37.871833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:37.871842 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:37.871896 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:37.872472 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:37.872960 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:37.874057 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:37.874278 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:37.874329 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:37.876442 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:37.876443 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:37.876444 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:37.876458 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:37.877920 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:37.878297 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:37.878298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:37.878299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:37.878361 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:37.878596 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:37.878596 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:37.878597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:37.878736 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:37.878800 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:37.879029 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 10:54:37.879118 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 10:54:37.879745 11956 heartbeater.cc:347] Connected to a master server at 127.10.238.254:44365
I20260929 10:54:37.879858 11956 heartbeater.cc:464] Registering TS with master...
I20260929 10:54:37.880041 11956 heartbeater.cc:511] Master 127.10.238.254:44365 requested a full tablet report, sending...
I20260929 10:54:37.880532 11765 ts_manager.cc:194] Registered new tserver with Master: 11dbb65fc43c4fa2b3e6a7d62131902d (127.10.238.193:39643)
I20260929 10:54:37.881139 11765 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.193@KRBTEST.COM'} at 127.10.238.193:45321
WARNING: no policy specified for kudu/127.10.238.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab.
Entry for principal kudu/127.10.238.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab.
I20260929 10:54:37.901722 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.194:0
--local_ip_for_outbound_sockets=127.10.238.194
--webserver_interface=127.10.238.194
--webserver_port=0
--tserver_master_addrs=127.10.238.254:44365
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab
--principal=kudu/127.10.238.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:40199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:54:37.987260 11961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:54:37.987453 11961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:54:37.987478 11961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:54:37.988559 11961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:54:37.988593 11961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:54:37.988619 11961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.194
I20260929 10:54:37.989997 11961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:40199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab
--principal=kudu/127.10.238.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.238.194
--webserver_port=0
--tserver_master_addrs=127.10.238.254:44365
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.194
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:54:37.990175 11961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:54:37.990342 11961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:54:37.995307 11967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:37.995442 11961 server_base.cc:1034] running on GCE node
W20260929 10:54:37.995328 11966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:37.995307 11969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:37.995764 11961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:54:37.995957 11961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:54:37.997094 11961 hybrid_clock.cc:648] HybridClock initialized: now 1790679277997079 us; error 34 us; skew 500 ppm
Sep 29 10:54:37 dist-test-slave-8nrt krb5kdc[11720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679277, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:54:37.999547 11961 init.cc:377] Logged in from keytab as kudu/127.10.238.194@KRBTEST.COM (short username kudu)
I20260929 10:54:37.999910 11961 webserver.cc:460] Webserver started at http://127.10.238.194:35387/ using document root <none> and password file <none>
I20260929 10:54:38.000082 11961 fs_manager.cc:359] Metadata directory not provided
I20260929 10:54:38.000121 11961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:54:38.000211 11961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:54:38.001163 11961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/ts-1/data/instance:
uuid: "c0617d4ccc4d4167aae085c7f4f81c37"
format_stamp: "Formatted at 2026-09-29 10:54:37 on dist-test-slave-8nrt"
server_key: "4a4c54f6a59ee3cdcf6941cfbbd40041"
server_key_iv: "a3d38c54996a6914f3ed0321798dc6fa"
server_key_version: "encryptionkey@0"
I20260929 10:54:38.001539 11961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/ts-1/wal/instance:
uuid: "c0617d4ccc4d4167aae085c7f4f81c37"
format_stamp: "Formatted at 2026-09-29 10:54:37 on dist-test-slave-8nrt"
server_key: "4a4c54f6a59ee3cdcf6941cfbbd40041"
server_key_iv: "a3d38c54996a6914f3ed0321798dc6fa"
server_key_version: "encryptionkey@0"
I20260929 10:54:38.002744 11961 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:54:38.003381 11976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:38.003571 11961 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:54:38.003638 11961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/ts-1/wal
uuid: "c0617d4ccc4d4167aae085c7f4f81c37"
format_stamp: "Formatted at 2026-09-29 10:54:37 on dist-test-slave-8nrt"
server_key: "4a4c54f6a59ee3cdcf6941cfbbd40041"
server_key_iv: "a3d38c54996a6914f3ed0321798dc6fa"
server_key_version: "encryptionkey@0"
I20260929 10:54:38.003726 11961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:54:38.034534 11961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:54:38.036677 11961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:54:38.036830 11961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:54:38.037070 11961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:54:38.037453 11961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:54:38.037506 11961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:38.037550 11961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:54:38.037577 11961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:38.043669 11961 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.194:41449
I20260929 10:54:38.043712 12089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.194:41449 every 8 connection(s)
I20260929 10:54:38.044023 11961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
I20260929 10:54:38.046386 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 11961
I20260929 10:54:38.046496 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/ts-1/wal/instance
I20260929 10:54:38.046665 11195 external_mini_cluster.cc:1428] Setting key 60667edc8fb4c9e7e5436be591fe2a6b
Sep 29 10:54:38 dist-test-slave-8nrt krb5kdc[11720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679277, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.194@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:54:38.054771 11804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:38.044873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:33833 (local address 127.10.238.254:44365)
0929 10:54:38.044921 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:38.044923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:38.045101 (+ 178us) server_negotiation.cc:407] Connection header received
0929 10:54:38.046143 (+ 1042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:38.046146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:38.046159 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:38.046221 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:38.047039 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:38.047616 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:38.048800 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:38.049086 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:38.049162 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:38.051692 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:38.051694 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:38.051695 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:38.051703 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:38.053400 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:38.053789 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:38.053791 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:38.053791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:38.053857 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:38.054094 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:38.054095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:38.054096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:38.054236 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:38.054325 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:38.054568 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 10:54:38.054668 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 10:54:38.054797 12092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:38.044823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:44365 (local address 127.10.238.194:33833)
0929 10:54:38.044971 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:38.045024 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:38.046029 (+ 1005us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:38.046330 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:38.046334 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:38.046655 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:54:38.046962 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:38.046969 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:38.047714 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:38.047716 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:38.048696 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:38.048699 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:38.049170 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:38.049171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:38.049340 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:38.049912 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:54:38.049920 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:54:38.051589 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:54:38.053487 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:38.053494 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:38.053496 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:38.053710 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:38.053931 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:38.053931 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:38.053932 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:38.054028 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:38.054317 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:54:38.054321 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:54:38.054434 (+ 113us) client_negotiation.cc:769] Sending connection context
0929 10:54:38.054516 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 10:54:38.054633 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":68,"threads_started":1}
I20260929 10:54:38.055327 12090 heartbeater.cc:347] Connected to a master server at 127.10.238.254:44365
I20260929 10:54:38.055431 12090 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.10.238.195@KRBTEST.COM; defaulting to no policy
I20260929 10:54:38.055650 12090 heartbeater.cc:511] Master 127.10.238.254:44365 requested a full tablet report, sending...
I20260929 10:54:38.056030 11765 ts_manager.cc:194] Registered new tserver with Master: c0617d4ccc4d4167aae085c7f4f81c37 (127.10.238.194:41449)
I20260929 10:54:38.056443 11765 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.194@KRBTEST.COM'} at 127.10.238.194:33833
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab.
Entry for principal kudu/127.10.238.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab.
I20260929 10:54:38.073822 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.195:0
--local_ip_for_outbound_sockets=127.10.238.195
--webserver_interface=127.10.238.195
--webserver_port=0
--tserver_master_addrs=127.10.238.254:44365
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab
--principal=kudu/127.10.238.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:40199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:54:38.154968 12095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:54:38.155191 12095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:54:38.155230 12095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:54:38.156394 12095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:54:38.156458 12095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:54:38.156505 12095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.195
I20260929 10:54:38.158061 12095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:40199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab
--principal=kudu/127.10.238.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.238.195
--webserver_port=0
--tserver_master_addrs=127.10.238.254:44365
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.195
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:54:38.158320 12095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:54:38.158548 12095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:54:38.163787 12100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:38.163787 12103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:38.163801 12101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:38.163838 12095 server_base.cc:1034] running on GCE node
I20260929 10:54:38.164160 12095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:54:38.164412 12095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:54:38.165580 12095 hybrid_clock.cc:648] HybridClock initialized: now 1790679278165562 us; error 32 us; skew 500 ppm
Sep 29 10:54:38 dist-test-slave-8nrt krb5kdc[11720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679278, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:54:38.168136 12095 init.cc:377] Logged in from keytab as kudu/127.10.238.195@KRBTEST.COM (short username kudu)
I20260929 10:54:38.168560 12095 webserver.cc:460] Webserver started at http://127.10.238.195:45387/ using document root <none> and password file <none>
I20260929 10:54:38.168795 12095 fs_manager.cc:359] Metadata directory not provided
I20260929 10:54:38.168865 12095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:54:38.169018 12095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:54:38.170096 12095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/ts-2/data/instance:
uuid: "60ebb9a9d502498bbe2e0e033162ef1d"
format_stamp: "Formatted at 2026-09-29 10:54:38 on dist-test-slave-8nrt"
server_key: "1a02c384b49933ff0ec59fbe1247ce55"
server_key_iv: "680a3ddfbd9a5ec991f04b5cef90c65b"
server_key_version: "encryptionkey@0"
I20260929 10:54:38.170464 12095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/ts-2/wal/instance:
uuid: "60ebb9a9d502498bbe2e0e033162ef1d"
format_stamp: "Formatted at 2026-09-29 10:54:38 on dist-test-slave-8nrt"
server_key: "1a02c384b49933ff0ec59fbe1247ce55"
server_key_iv: "680a3ddfbd9a5ec991f04b5cef90c65b"
server_key_version: "encryptionkey@0"
I20260929 10:54:38.171783 12095 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 10:54:38.172511 12110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:38.172685 12095 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:54:38.172791 12095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/ts-2/wal
uuid: "60ebb9a9d502498bbe2e0e033162ef1d"
format_stamp: "Formatted at 2026-09-29 10:54:38 on dist-test-slave-8nrt"
server_key: "1a02c384b49933ff0ec59fbe1247ce55"
server_key_iv: "680a3ddfbd9a5ec991f04b5cef90c65b"
server_key_version: "encryptionkey@0"
I20260929 10:54:38.172881 12095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:54:38.185688 12095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:54:38.187964 12095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:54:38.188126 12095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:54:38.188369 12095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:54:38.188679 12095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:54:38.188735 12095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:38.188776 12095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:54:38.188803 12095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:38.194890 12095 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.195:44413
I20260929 10:54:38.194931 12223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.195:44413 every 8 connection(s)
I20260929 10:54:38.195274 12095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
I20260929 10:54:38.198460 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 12095
I20260929 10:54:38.198567 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790679274510869-11195-0/minicluster-data/ts-2/wal/instance
I20260929 10:54:38.198742 11195 external_mini_cluster.cc:1428] Setting key 3028e9ae9eb319d524efb594386de47f
Sep 29 10:54:38 dist-test-slave-8nrt krb5kdc[11720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679278, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.195@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:54:38.204721 11804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:38.195988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:53671 (local address 127.10.238.254:44365)
0929 10:54:38.196038 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:38.196040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:38.196288 (+ 248us) server_negotiation.cc:407] Connection header received
0929 10:54:38.197195 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:38.197196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:38.197206 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:38.197313 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:38.197878 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:38.198367 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:38.199814 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:38.200004 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:38.200055 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:38.202406 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:38.202407 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:38.202408 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:38.202414 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:38.203612 (+ 1198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:38.203981 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:38.203982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:38.203982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:38.204030 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:38.204229 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:38.204230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:38.204230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:38.204334 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:38.204383 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:38.204601 (+ 218us) server_negotiation.cc:299] Negotiation successful
0929 10:54:38.204657 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 10:54:38.204923 12226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:38.195972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:44365 (local address 127.10.238.195:53671)
0929 10:54:38.196149 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:38.196183 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:38.197104 (+ 921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:38.197383 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:38.197386 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:38.197622 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:54:38.197804 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:38.197807 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:38.198460 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:38.198461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:38.199747 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:38.199750 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:38.200073 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:38.200073 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:38.200227 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:38.200728 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:54:38.200736 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:54:38.202317 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:54:38.203681 (+ 1364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:38.203688 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:38.203689 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:38.203917 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:38.204085 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:38.204085 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:38.204086 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:38.204178 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:38.204390 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:54:38.204395 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:54:38.204500 (+ 105us) client_negotiation.cc:769] Sending connection context
0929 10:54:38.204583 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 10:54:38.204696 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":97,"threads_started":1}
I20260929 10:54:38.205437 12224 heartbeater.cc:347] Connected to a master server at 127.10.238.254:44365
I20260929 10:54:38.205549 12224 heartbeater.cc:464] Registering TS with master...
I20260929 10:54:38.205724 12224 heartbeater.cc:511] Master 127.10.238.254:44365 requested a full tablet report, sending...
I20260929 10:54:38.206120 11765 ts_manager.cc:194] Registered new tserver with Master: 60ebb9a9d502498bbe2e0e033162ef1d (127.10.238.195:44413)
I20260929 10:54:38.206516 11765 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.195@KRBTEST.COM'} at 127.10.238.195:53671
I20260929 10:54:38.210417 11195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:54:38.217973 11804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:38.211835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42076 (local address 127.10.238.254:44365)
0929 10:54:38.211914 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:38.211916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:38.212039 (+ 123us) server_negotiation.cc:407] Connection header received
0929 10:54:38.212114 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:38.212115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:38.212125 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:38.212174 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:38.212697 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:38.213168 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:38.213905 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:38.214126 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:38.214198 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:38.215390 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:38.215391 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:38.215392 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:38.215399 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:38.216786 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:38.217148 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:38.217149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:38.217150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:38.217209 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:38.217438 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:38.217439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:38.217440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:38.217586 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:38.217691 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:38.217809 (+ 118us) server_negotiation.cc:299] Negotiation successful
0929 10:54:38.217878 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 10:54:38.220085 11765 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42076:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:54:38.221665 11765 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:54:38.228004 12238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:38.224003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49126 (local address 127.10.238.195:44413)
0929 10:54:38.224166 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:38.224171 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:38.224183 (+ 12us) server_negotiation.cc:407] Connection header received
0929 10:54:38.224223 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:38.224225 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:38.224242 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:38.224352 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:54:38.224481 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:38.225336 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:38.226663 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:38.227535 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:38.227633 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:38.227706 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:38.227869 (+ 163us) server_negotiation.cc:299] Negotiation successful
0929 10:54:38.227911 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":72,"threads_started":1}
I20260929 10:54:38.228035 12235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:38.223859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.195:44413 (local address 127.0.0.1:49126)
0929 10:54:38.224018 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:38.224056 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:38.224118 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:38.224357 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:38.224360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:38.224362 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:54:38.224425 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:38.224426 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:38.225434 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:38.225435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:38.226465 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:38.226467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:38.227633 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:38.227634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:38.227735 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:38.227750 (+ 15us) client_negotiation.cc:769] Sending connection context
0929 10:54:38.227850 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 10:54:38.227911 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":62,"threads_started":1}
I20260929 10:54:38.228204 12239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:38.224042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57250 (local address 127.10.238.194:41449)
0929 10:54:38.224225 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:38.224229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:38.224508 (+ 279us) server_negotiation.cc:407] Connection header received
0929 10:54:38.224552 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:38.224555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:38.224571 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:38.224662 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:54:38.224868 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:38.225933 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:38.227004 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:38.227621 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:38.227686 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:38.227732 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:38.228078 (+ 346us) server_negotiation.cc:299] Negotiation successful
0929 10:54:38.228123 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":78,"threads_started":1}
I20260929 10:54:38.228039 12236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:38.223860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.194:41449 (local address 127.0.0.1:57250)
0929 10:54:38.224275 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:38.224286 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:38.224341 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:38.224760 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:38.224760 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:38.224762 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:54:38.224791 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:38.224792 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:38.226039 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:38.226040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:38.226922 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:38.226924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:38.227778 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:38.227779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:38.227838 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:38.227842 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 10:54:38.227891 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 10:54:38.227937 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"mutex_wait_us":4,"thread_start_us":78,"threads_started":1}
I20260929 10:54:38.228446 12237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:38.223877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:39643 (local address 127.0.0.1:44370)
0929 10:54:38.224295 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:38.224304 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:38.224354 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:38.224639 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:38.224640 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:38.224643 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:54:38.224688 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:38.224690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:38.225991 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:38.225992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:38.227223 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:38.227227 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:38.228196 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:38.228197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:38.228273 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:38.228277 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 10:54:38.228331 (+ 54us) client_negotiation.cc:240] Negotiation successful
0929 10:54:38.228376 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":1,"thread_start_us":75,"threads_started":1}
I20260929 10:54:38.228682 12240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:38.224116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44370 (local address 127.10.238.193:39643)
0929 10:54:38.224282 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:38.224287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:38.224409 (+ 122us) server_negotiation.cc:407] Connection header received
0929 10:54:38.224451 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:38.224453 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:38.224470 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:38.224542 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:54:38.224774 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:38.225877 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:38.227349 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:38.228050 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:38.228116 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:38.228163 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:38.228562 (+ 399us) server_negotiation.cc:299] Negotiation successful
0929 10:54:38.228608 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":85,"threads_started":1}
I20260929 10:54:38.229045 12024 tablet_service.cc:1467] Processing CreateTablet for tablet d9487b58caab4e05adafb90a769bc392 (DEFAULT_TABLE table=test-table [id=7717e68af54441ff8d276878957b5473]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:54:38.229481 12024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9487b58caab4e05adafb90a769bc392. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:54:38.229493 11890 tablet_service.cc:1467] Processing CreateTablet for tablet d9487b58caab4e05adafb90a769bc392 (DEFAULT_TABLE table=test-table [id=7717e68af54441ff8d276878957b5473]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:54:38.229840 11890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9487b58caab4e05adafb90a769bc392. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:54:38.229830 12158 tablet_service.cc:1467] Processing CreateTablet for tablet d9487b58caab4e05adafb90a769bc392 (DEFAULT_TABLE table=test-table [id=7717e68af54441ff8d276878957b5473]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:54:38.230067 12158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9487b58caab4e05adafb90a769bc392. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:54:38.232594 12241 tablet_bootstrap.cc:492] T d9487b58caab4e05adafb90a769bc392 P 60ebb9a9d502498bbe2e0e033162ef1d: Bootstrap starting.
I20260929 10:54:38.233300 12241 tablet_bootstrap.cc:654] T d9487b58caab4e05adafb90a769bc392 P 60ebb9a9d502498bbe2e0e033162ef1d: Neither blocks nor log segments found. Creating new log.
I20260929 10:54:38.233539 12243 tablet_bootstrap.cc:492] T d9487b58caab4e05adafb90a769bc392 P 11dbb65fc43c4fa2b3e6a7d62131902d: Bootstrap starting.
I20260929 10:54:38.233682 12242 tablet_bootstrap.cc:492] T d9487b58caab4e05adafb90a769bc392 P c0617d4ccc4d4167aae085c7f4f81c37: Bootstrap starting.
I20260929 10:54:38.233949 12241 log.cc:824] T d9487b58caab4e05adafb90a769bc392 P 60ebb9a9d502498bbe2e0e033162ef1d: Log is configured to *not* fsync() on all Append() calls
I20260929 10:54:38.234241 12243 tablet_bootstrap.cc:654] T d9487b58caab4e05adafb90a769bc392 P 11dbb65fc43c4fa2b3e6a7d62131902d: Neither blocks nor log segments found. Creating new log.
I20260929 10:54:38.234395 12242 tablet_bootstrap.cc:654] T d9487b58caab4e05adafb90a769bc392 P c0617d4ccc4d4167aae085c7f4f81c37: Neither blocks nor log segments found. Creating new log.
I20260929 10:54:38.234807 12241 tablet_bootstrap.cc:492] T d9487b58caab4e05adafb90a769bc392 P 60ebb9a9d502498bbe2e0e033162ef1d: No bootstrap required, opened a new log
I20260929 10:54:38.234820 12243 log.cc:824] T d9487b58caab4e05adafb90a769bc392 P 11dbb65fc43c4fa2b3e6a7d62131902d: Log is configured to *not* fsync() on all Append() calls
I20260929 10:54:38.234880 12241 ts_tablet_manager.cc:1397] T d9487b58caab4e05adafb90a769bc392 P 60ebb9a9d502498bbe2e0e033162ef1d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 10:54:38.234962 12242 log.cc:824] T d9487b58caab4e05adafb90a769bc392 P c0617d4ccc4d4167aae085c7f4f81c37: Log is configured to *not* fsync() on all Append() calls
I20260929 10:54:38.235631 12243 tablet_bootstrap.cc:492] T d9487b58caab4e05adafb90a769bc392 P 11dbb65fc43c4fa2b3e6a7d62131902d: No bootstrap required, opened a new log
I20260929 10:54:38.235656 12242 tablet_bootstrap.cc:492] T d9487b58caab4e05adafb90a769bc392 P c0617d4ccc4d4167aae085c7f4f81c37: No bootstrap required, opened a new log
I20260929 10:54:38.235702 12243 ts_tablet_manager.cc:1397] T d9487b58caab4e05adafb90a769bc392 P 11dbb65fc43c4fa2b3e6a7d62131902d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 10:54:38.235744 12242 ts_tablet_manager.cc:1397] T d9487b58caab4e05adafb90a769bc392 P c0617d4ccc4d4167aae085c7f4f81c37: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 10:54:38.237039 12241 raft_consensus.cc:348] T d9487b58caab4e05adafb90a769bc392 P 60ebb9a9d502498bbe2e0e033162ef1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ebb9a9d502498bbe2e0e033162ef1d" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 44413 } } peers { permanent_uuid: "c0617d4ccc4d4167aae085c7f4f81c37" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 41449 } } peers { permanent_uuid: "11dbb65fc43c4fa2b3e6a7d62131902d" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 39643 } }
I20260929 10:54:38.237223 12241 raft_consensus.cc:374] T d9487b58caab4e05adafb90a769bc392 P 60ebb9a9d502498bbe2e0e033162ef1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:54:38.237310 12241 raft_consensus.cc:729] T d9487b58caab4e05adafb90a769bc392 P 60ebb9a9d502498bbe2e0e033162ef1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ebb9a9d502498bbe2e0e033162ef1d, State: Initialized, Role: FOLLOWER
I20260929 10:54:38.237260 12242 raft_consensus.cc:348] T d9487b58caab4e05adafb90a769bc392 P c0617d4ccc4d4167aae085c7f4f81c37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ebb9a9d502498bbe2e0e033162ef1d" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 44413 } } peers { permanent_uuid: "c0617d4ccc4d4167aae085c7f4f81c37" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 41449 } } peers { permanent_uuid: "11dbb65fc43c4fa2b3e6a7d62131902d" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 39643 } }
I20260929 10:54:38.237433 12241 consensus_queue.cc:260] T d9487b58caab4e05adafb90a769bc392 P 60ebb9a9d502498bbe2e0e033162ef1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ebb9a9d502498bbe2e0e033162ef1d" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 44413 } } peers { permanent_uuid: "c0617d4ccc4d4167aae085c7f4f81c37" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 41449 } } peers { permanent_uuid: "11dbb65fc43c4fa2b3e6a7d62131902d" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 39643 } }
I20260929 10:54:38.237500 12242 raft_consensus.cc:374] T d9487b58caab4e05adafb90a769bc392 P c0617d4ccc4d4167aae085c7f4f81c37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:54:38.237555 12242 raft_consensus.cc:729] T d9487b58caab4e05adafb90a769bc392 P c0617d4ccc4d4167aae085c7f4f81c37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0617d4ccc4d4167aae085c7f4f81c37, State: Initialized, Role: FOLLOWER
I20260929 10:54:38.237705 12241 ts_tablet_manager.cc:1428] T d9487b58caab4e05adafb90a769bc392 P 60ebb9a9d502498bbe2e0e033162ef1d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 10:54:38.237670 12242 consensus_queue.cc:260] T d9487b58caab4e05adafb90a769bc392 P c0617d4ccc4d4167aae085c7f4f81c37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ebb9a9d502498bbe2e0e033162ef1d" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 44413 } } peers { permanent_uuid: "c0617d4ccc4d4167aae085c7f4f81c37" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 41449 } } peers { permanent_uuid: "11dbb65fc43c4fa2b3e6a7d62131902d" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 39643 } }
I20260929 10:54:38.237767 12224 heartbeater.cc:503] Master 127.10.238.254:44365 was elected leader, sending a full tablet report...
I20260929 10:54:38.237732 12243 raft_consensus.cc:348] T d9487b58caab4e05adafb90a769bc392 P 11dbb65fc43c4fa2b3e6a7d62131902d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ebb9a9d502498bbe2e0e033162ef1d" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 44413 } } peers { permanent_uuid: "c0617d4ccc4d4167aae085c7f4f81c37" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 41449 } } peers { permanent_uuid: "11dbb65fc43c4fa2b3e6a7d62131902d" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 39643 } }
I20260929 10:54:38.237893 12243 raft_consensus.cc:374] T d9487b58caab4e05adafb90a769bc392 P 11dbb65fc43c4fa2b3e6a7d62131902d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:54:38.237933 12243 raft_consensus.cc:729] T d9487b58caab4e05adafb90a769bc392 P 11dbb65fc43c4fa2b3e6a7d62131902d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11dbb65fc43c4fa2b3e6a7d62131902d, State: Initialized, Role: FOLLOWER
I20260929 10:54:38.237970 12242 ts_tablet_manager.cc:1428] T d9487b58caab4e05adafb90a769bc392 P c0617d4ccc4d4167aae085c7f4f81c37: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 10:54:38.237996 12090 heartbeater.cc:503] Master 127.10.238.254:44365 was elected leader, sending a full tablet report...
I20260929 10:54:38.238035 12243 consensus_queue.cc:260] T d9487b58caab4e05adafb90a769bc392 P 11dbb65fc43c4fa2b3e6a7d62131902d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ebb9a9d502498bbe2e0e033162ef1d" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 44413 } } peers { permanent_uuid: "c0617d4ccc4d4167aae085c7f4f81c37" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 41449 } } peers { permanent_uuid: "11dbb65fc43c4fa2b3e6a7d62131902d" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 39643 } }
I20260929 10:54:38.238296 12243 ts_tablet_manager.cc:1428] T d9487b58caab4e05adafb90a769bc392 P 11dbb65fc43c4fa2b3e6a7d62131902d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 10:54:38.238441 11956 heartbeater.cc:503] Master 127.10.238.254:44365 was elected leader, sending a full tablet report...
I20260929 10:54:38.275540 12248 raft_consensus.cc:482] T d9487b58caab4e05adafb90a769bc392 P c0617d4ccc4d4167aae085c7f4f81c37 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:54:38.275676 12248 raft_consensus.cc:504] T d9487b58caab4e05adafb90a769bc392 P c0617d4ccc4d4167aae085c7f4f81c37 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ebb9a9d502498bbe2e0e033162ef1d" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 44413 } } peers { permanent_uuid: "c0617d4ccc4d4167aae085c7f4f81c37" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 41449 } } peers { permanent_uuid: "11dbb65fc43c4fa2b3e6a7d62131902d" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 39643 } }
I20260929 10:54:38.276068 12248 leader_election.cc:288] T d9487b58caab4e05adafb90a769bc392 P c0617d4ccc4d4167aae085c7f4f81c37 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60ebb9a9d502498bbe2e0e033162ef1d (127.10.238.195:44413), 11dbb65fc43c4fa2b3e6a7d62131902d (127.10.238.193:39643)
I20260929 10:54:38.279996 12238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:38.276319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:43573 (local address 127.10.238.195:44413)
0929 10:54:38.276412 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:38.276414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:38.276425 (+ 11us) server_negotiation.cc:407] Connection header received
0929 10:54:38.276453 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:38.276454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:38.276471 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:38.276524 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:54:38.276716 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:38.277523 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:38.279080 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:38.279652 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:38.279717 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:38.279728 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:38.279886 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 10:54:38.279926 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 10:54:38.279996 12240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:38.276330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:36609 (local address 127.10.238.193:39643)
0929 10:54:38.276420 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:38.276422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:38.276557 (+ 135us) server_negotiation.cc:407] Connection header received
0929 10:54:38.276600 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:38.276602 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:38.276617 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:38.276654 (+ 37us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:54:38.276794 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:38.277517 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:38.279080 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:38.279638 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:38.279740 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:38.279748 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:38.279886 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 10:54:38.279927 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 10:54:38.280017 12092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:38.276242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.195:44413 (local address 127.10.238.194:43573)
0929 10:54:38.276311 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:38.276325 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:38.276379 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:38.276549 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:38.276550 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:38.276552 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:54:38.276633 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:38.276634 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:38.277609 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:38.277611 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:38.278984 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:38.278988 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:38.279726 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:38.279727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:38.279806 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:38.279810 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 10:54:38.279867 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 10:54:38.279916 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":100}
I20260929 10:54:38.280025 12250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:38.276242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:39643 (local address 127.10.238.194:36609)
0929 10:54:38.276464 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:38.276475 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:38.276547 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:38.276662 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:38.276662 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:38.276663 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:54:38.276728 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:38.276729 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:38.277609 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:38.277611 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:38.278984 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:38.278988 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:38.279726 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:38.279727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:38.279806 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:38.279810 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 10:54:38.279867 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 10:54:38.279916 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":82,"thread_start_us":118,"threads_started":1}
I20260929 10:54:38.280404 12178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9487b58caab4e05adafb90a769bc392" candidate_uuid: "c0617d4ccc4d4167aae085c7f4f81c37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ebb9a9d502498bbe2e0e033162ef1d" is_pre_election: true
I20260929 10:54:38.280462 11910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9487b58caab4e05adafb90a769bc392" candidate_uuid: "c0617d4ccc4d4167aae085c7f4f81c37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11dbb65fc43c4fa2b3e6a7d62131902d" is_pre_election: true
I20260929 10:54:38.280568 12178 raft_consensus.cc:2454] T d9487b58caab4e05adafb90a769bc392 P 60ebb9a9d502498bbe2e0e033162ef1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0617d4ccc4d4167aae085c7f4f81c37 in term 0.
I20260929 10:54:38.280576 11910 raft_consensus.cc:2454] T d9487b58caab4e05adafb90a769bc392 P 11dbb65fc43c4fa2b3e6a7d62131902d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0617d4ccc4d4167aae085c7f4f81c37 in term 0.
I20260929 10:54:38.280799 11980 leader_election.cc:302] T d9487b58caab4e05adafb90a769bc392 P c0617d4ccc4d4167aae085c7f4f81c37 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11dbb65fc43c4fa2b3e6a7d62131902d, c0617d4ccc4d4167aae085c7f4f81c37; no voters:
I20260929 10:54:38.280925 12248 raft_consensus.cc:2781] T d9487b58caab4e05adafb90a769bc392 P c0617d4ccc4d4167aae085c7f4f81c37 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:54:38.280997 12248 raft_consensus.cc:482] T d9487b58caab4e05adafb90a769bc392 P c0617d4ccc4d4167aae085c7f4f81c37 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:54:38.281035 12248 raft_consensus.cc:3037] T d9487b58caab4e05adafb90a769bc392 P c0617d4ccc4d4167aae085c7f4f81c37 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:54:38.281765 12248 raft_consensus.cc:504] T d9487b58caab4e05adafb90a769bc392 P c0617d4ccc4d4167aae085c7f4f81c37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ebb9a9d502498bbe2e0e033162ef1d" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 44413 } } peers { permanent_uuid: "c0617d4ccc4d4167aae085c7f4f81c37" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 41449 } } peers { permanent_uuid: "11dbb65fc43c4fa2b3e6a7d62131902d" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 39643 } }
I20260929 10:54:38.281915 12248 leader_election.cc:288] T d9487b58caab4e05adafb90a769bc392 P c0617d4ccc4d4167aae085c7f4f81c37 [CANDIDATE]: Term 1 election: Requested vote from peers 60ebb9a9d502498bbe2e0e033162ef1d (127.10.238.195:44413), 11dbb65fc43c4fa2b3e6a7d62131902d (127.10.238.193:39643)
I20260929 10:54:38.282048 11910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9487b58caab4e05adafb90a769bc392" candidate_uuid: "c0617d4ccc4d4167aae085c7f4f81c37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11dbb65fc43c4fa2b3e6a7d62131902d"
I20260929 10:54:38.282048 12178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9487b58caab4e05adafb90a769bc392" candidate_uuid: "c0617d4ccc4d4167aae085c7f4f81c37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ebb9a9d502498bbe2e0e033162ef1d"
I20260929 10:54:38.282127 12178 raft_consensus.cc:3037] T d9487b58caab4e05adafb90a769bc392 P 60ebb9a9d502498bbe2e0e033162ef1d [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:54:38.282127 11910 raft_consensus.cc:3037] T d9487b58caab4e05adafb90a769bc392 P 11dbb65fc43c4fa2b3e6a7d62131902d [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:54:38.283013 11910 raft_consensus.cc:2454] T d9487b58caab4e05adafb90a769bc392 P 11dbb65fc43c4fa2b3e6a7d62131902d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0617d4ccc4d4167aae085c7f4f81c37 in term 1.
I20260929 10:54:38.283013 12178 raft_consensus.cc:2454] T d9487b58caab4e05adafb90a769bc392 P 60ebb9a9d502498bbe2e0e033162ef1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0617d4ccc4d4167aae085c7f4f81c37 in term 1.
I20260929 10:54:38.283180 11980 leader_election.cc:302] T d9487b58caab4e05adafb90a769bc392 P c0617d4ccc4d4167aae085c7f4f81c37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11dbb65fc43c4fa2b3e6a7d62131902d, c0617d4ccc4d4167aae085c7f4f81c37; no voters:
I20260929 10:54:38.283277 12248 raft_consensus.cc:2781] T d9487b58caab4e05adafb90a769bc392 P c0617d4ccc4d4167aae085c7f4f81c37 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:54:38.283430 12248 raft_consensus.cc:686] T d9487b58caab4e05adafb90a769bc392 P c0617d4ccc4d4167aae085c7f4f81c37 [term 1 LEADER]: Becoming Leader. State: Replica: c0617d4ccc4d4167aae085c7f4f81c37, State: Running, Role: LEADER
I20260929 10:54:38.283531 12248 consensus_queue.cc:237] T d9487b58caab4e05adafb90a769bc392 P c0617d4ccc4d4167aae085c7f4f81c37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ebb9a9d502498bbe2e0e033162ef1d" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 44413 } } peers { permanent_uuid: "c0617d4ccc4d4167aae085c7f4f81c37" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 41449 } } peers { permanent_uuid: "11dbb65fc43c4fa2b3e6a7d62131902d" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 39643 } }
I20260929 10:54:38.284368 11764 catalog_manager.cc:5495] T d9487b58caab4e05adafb90a769bc392 P c0617d4ccc4d4167aae085c7f4f81c37 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0617d4ccc4d4167aae085c7f4f81c37 (127.10.238.194). New cstate: current_term: 1 leader_uuid: "c0617d4ccc4d4167aae085c7f4f81c37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ebb9a9d502498bbe2e0e033162ef1d" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 44413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0617d4ccc4d4167aae085c7f4f81c37" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 41449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11dbb65fc43c4fa2b3e6a7d62131902d" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 39643 } health_report { overall_health: UNKNOWN } } }
W20260929 10:54:38.294536 12091 tablet.cc:2369] T d9487b58caab4e05adafb90a769bc392 P c0617d4ccc4d4167aae085c7f4f81c37: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 10:54:38 dist-test-slave-8nrt krb5kdc[11720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.238.193@KRBTEST.COM
I20260929 10:54:38.303712 12240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:38.296844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44384 (local address 127.10.238.193:39643)
0929 10:54:38.296922 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:38.296923 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:38.296975 (+ 52us) server_negotiation.cc:407] Connection header received
0929 10:54:38.297016 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:38.297017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:38.297026 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:38.297072 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:38.297648 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:38.298116 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:38.298803 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:38.298974 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:38.299021 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:38.301316 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:38.301325 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:38.301325 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:38.301332 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:38.302746 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:38.303060 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:38.303063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:38.303063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:38.303116 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:38.303283 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:38.303284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:38.303284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:38.303414 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:38.303524 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:38.303558 (+ 34us) server_negotiation.cc:299] Negotiation successful
0929 10:54:38.303616 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Sep 29 10:54:38 dist-test-slave-8nrt krb5kdc[11720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679278, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 10:54:38 dist-test-slave-8nrt krb5kdc[11720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679278, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.238.193@KRBTEST.COM
I20260929 10:54:38.326591 12240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:38.319365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44394 (local address 127.10.238.193:39643)
0929 10:54:38.319442 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:38.319445 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:38.319555 (+ 110us) server_negotiation.cc:407] Connection header received
0929 10:54:38.319652 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:38.319653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:38.319670 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:38.319736 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:38.320325 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:38.320795 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:38.321567 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:38.321762 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:38.321838 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:38.324121 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:38.324122 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:38.324122 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:38.324132 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:38.325425 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:38.325750 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:38.325751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:38.325752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:38.325814 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:38.326006 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:38.326007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:38.326008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:38.326113 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:38.326208 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:38.326407 (+ 199us) server_negotiation.cc:299] Negotiation successful
0929 10:54:38.326487 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Sep 29 10:54:38 dist-test-slave-8nrt krb5kdc[11720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 10:54:38 dist-test-slave-8nrt krb5kdc[11720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.238.193@KRBTEST.COM
I20260929 10:54:38.349063 12240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:38.341816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44410 (local address 127.10.238.193:39643)
0929 10:54:38.341883 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:38.341886 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:38.341965 (+ 79us) server_negotiation.cc:407] Connection header received
0929 10:54:38.342059 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:38.342060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:38.342076 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:38.342136 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:38.342732 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:38.343226 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:38.343960 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:38.344151 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:38.344235 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:38.346490 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:38.346492 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:38.346492 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:38.346498 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:38.348001 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:38.348308 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:38.348309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:38.348309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:38.348373 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:38.348563 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:38.348563 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:38.348564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:38.348691 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:38.348767 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:38.348923 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 10:54:38.348990 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260929 10:54:38.349612 11870 tablet_service.cc:2908] Rejecting scan request for tablet d9487b58caab4e05adafb90a769bc392: Uninitialized: clean time has not yet been initialized
I20260929 10:54:38.350447 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 11827
I20260929 10:54:38.355497 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 11961
I20260929 10:54:38.360513 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 12095
I20260929 10:54:38.365128 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 11733
2026-09-29T10:54:38Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1969 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:54:38 dist-test-slave-8nrt krb5kdc[12265](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:54:38 dist-test-slave-8nrt krb5kdc[12265](info): set up 2 sockets
Sep 29 10:54:38 dist-test-slave-8nrt krb5kdc[12265](info): commencing operation
krb5kdc: starting...
W20260929 10:54:39.400815 11195 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Sep 29 10:54:39 dist-test-slave-8nrt krb5kdc[12265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:54:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:54:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
I20260929 10:54:39.517788 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:41855
--webserver_interface=127.10.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:37781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:41855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:54:39.596704 12278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:54:39.596975 12278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:54:39.597013 12278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:54:39.598177 12278 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:54:39.598248 12278 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 10:54:39.598277 12278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:54:39.598306 12278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:54:39.598336 12278 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:54:39.598362 12278 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:54:39.599812 12278 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:37781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:41855
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:41855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:54:39.600086 12278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:54:39.600317 12278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:54:39.605077 12286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:39.605086 12284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:39.605086 12283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:39.605294 12278 server_base.cc:1034] running on GCE node
I20260929 10:54:39.605575 12278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:54:39.605897 12278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:54:39.607057 12278 hybrid_clock.cc:648] HybridClock initialized: now 1790679279607044 us; error 41 us; skew 500 ppm
Sep 29 10:54:39 dist-test-slave-8nrt krb5kdc[12265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679279, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:54:39.609524 12278 init.cc:377] Logged in from keytab as kudu/127.10.238.254@KRBTEST.COM (short username kudu)
I20260929 10:54:39.609894 12278 webserver.cc:460] Webserver started at http://127.10.238.254:38157/ using document root <none> and password file <none>
I20260929 10:54:39.610065 12278 fs_manager.cc:359] Metadata directory not provided
I20260929 10:54:39.610102 12278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:54:39.610185 12278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:54:39.611160 12278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/master-0/data/instance:
uuid: "c26351720c0f48109cd9a0bb30c62f84"
format_stamp: "Formatted at 2026-09-29 10:54:39 on dist-test-slave-8nrt"
server_key: "3010cfb16f70c4b6deeed09f0c066444"
server_key_iv: "a7e08ac11f9beb8f4406ed782f2b748c"
server_key_version: "encryptionkey@0"
I20260929 10:54:39.611466 12278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/master-0/wal/instance:
uuid: "c26351720c0f48109cd9a0bb30c62f84"
format_stamp: "Formatted at 2026-09-29 10:54:39 on dist-test-slave-8nrt"
server_key: "3010cfb16f70c4b6deeed09f0c066444"
server_key_iv: "a7e08ac11f9beb8f4406ed782f2b748c"
server_key_version: "encryptionkey@0"
I20260929 10:54:39.612759 12278 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260929 10:54:39.613440 12293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:39.613610 12278 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:39.613693 12278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "c26351720c0f48109cd9a0bb30c62f84"
format_stamp: "Formatted at 2026-09-29 10:54:39 on dist-test-slave-8nrt"
server_key: "3010cfb16f70c4b6deeed09f0c066444"
server_key_iv: "a7e08ac11f9beb8f4406ed782f2b748c"
server_key_version: "encryptionkey@0"
I20260929 10:54:39.613747 12278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:54:39.654175 12278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:54:39.656061 12278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:54:39.656172 12278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:54:39.660820 12278 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.254:41855
I20260929 10:54:39.660933 12355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.254:41855 every 8 connection(s)
I20260929 10:54:39.661295 12278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:54:39.662014 12356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:54:39.662504 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 12278
I20260929 10:54:39.662597 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/master-0/wal/instance
I20260929 10:54:39.662765 11195 external_mini_cluster.cc:1428] Setting key 1a3ae59b455aee9cf4c4fab5262c4e6e
I20260929 10:54:39.664693 12356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c26351720c0f48109cd9a0bb30c62f84: Bootstrap starting.
I20260929 10:54:39.665349 12356 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c26351720c0f48109cd9a0bb30c62f84: Neither blocks nor log segments found. Creating new log.
I20260929 10:54:39.665635 12356 log.cc:824] T 00000000000000000000000000000000 P c26351720c0f48109cd9a0bb30c62f84: Log is configured to *not* fsync() on all Append() calls
I20260929 10:54:39.666296 12356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c26351720c0f48109cd9a0bb30c62f84: No bootstrap required, opened a new log
I20260929 10:54:39.668279 12356 raft_consensus.cc:348] T 00000000000000000000000000000000 P c26351720c0f48109cd9a0bb30c62f84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c26351720c0f48109cd9a0bb30c62f84" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 41855 } }
I20260929 10:54:39.668437 12356 raft_consensus.cc:374] T 00000000000000000000000000000000 P c26351720c0f48109cd9a0bb30c62f84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:54:39.668497 12356 raft_consensus.cc:729] T 00000000000000000000000000000000 P c26351720c0f48109cd9a0bb30c62f84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c26351720c0f48109cd9a0bb30c62f84, State: Initialized, Role: FOLLOWER
I20260929 10:54:39.668660 12356 consensus_queue.cc:260] T 00000000000000000000000000000000 P c26351720c0f48109cd9a0bb30c62f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c26351720c0f48109cd9a0bb30c62f84" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 41855 } }
I20260929 10:54:39.668753 12356 raft_consensus.cc:388] T 00000000000000000000000000000000 P c26351720c0f48109cd9a0bb30c62f84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:54:39.668798 12356 raft_consensus.cc:482] T 00000000000000000000000000000000 P c26351720c0f48109cd9a0bb30c62f84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:54:39.668848 12356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c26351720c0f48109cd9a0bb30c62f84 [term 0 FOLLOWER]: Advancing to term 1
Sep 29 10:54:39 dist-test-slave-8nrt krb5kdc[12265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:54:39.669502 12356 raft_consensus.cc:504] T 00000000000000000000000000000000 P c26351720c0f48109cd9a0bb30c62f84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c26351720c0f48109cd9a0bb30c62f84" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 41855 } }
I20260929 10:54:39.669617 12356 leader_election.cc:302] T 00000000000000000000000000000000 P c26351720c0f48109cd9a0bb30c62f84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c26351720c0f48109cd9a0bb30c62f84; no voters:
I20260929 10:54:39.669802 12356 leader_election.cc:288] T 00000000000000000000000000000000 P c26351720c0f48109cd9a0bb30c62f84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:54:39.670070 12356 sys_catalog.cc:564] T 00000000000000000000000000000000 P c26351720c0f48109cd9a0bb30c62f84 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:54:39.669845 12361 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c26351720c0f48109cd9a0bb30c62f84 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:54:39.670289 12361 raft_consensus.cc:686] T 00000000000000000000000000000000 P c26351720c0f48109cd9a0bb30c62f84 [term 1 LEADER]: Becoming Leader. State: Replica: c26351720c0f48109cd9a0bb30c62f84, State: Running, Role: LEADER
I20260929 10:54:39.670370 12361 consensus_queue.cc:237] T 00000000000000000000000000000000 P c26351720c0f48109cd9a0bb30c62f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c26351720c0f48109cd9a0bb30c62f84" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 41855 } }
I20260929 10:54:39.671285 12363 sys_catalog.cc:455] T 00000000000000000000000000000000 P c26351720c0f48109cd9a0bb30c62f84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c26351720c0f48109cd9a0bb30c62f84. Latest consensus state: current_term: 1 leader_uuid: "c26351720c0f48109cd9a0bb30c62f84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c26351720c0f48109cd9a0bb30c62f84" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 41855 } } }
I20260929 10:54:39.671396 12363 sys_catalog.cc:458] T 00000000000000000000000000000000 P c26351720c0f48109cd9a0bb30c62f84 [sys.catalog]: This master's current role is: LEADER
I20260929 10:54:39.672164 12362 sys_catalog.cc:455] T 00000000000000000000000000000000 P c26351720c0f48109cd9a0bb30c62f84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c26351720c0f48109cd9a0bb30c62f84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c26351720c0f48109cd9a0bb30c62f84" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 41855 } } }
I20260929 10:54:39.672290 12362 sys_catalog.cc:458] T 00000000000000000000000000000000 P c26351720c0f48109cd9a0bb30c62f84 [sys.catalog]: This master's current role is: LEADER
I20260929 10:54:39.672639 12373 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:54:39.673132 12373 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:54:39.673599 12359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:39.663300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38614 (local address 127.10.238.254:41855)
0929 10:54:39.663490 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:39.663498 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:39.663522 (+ 24us) server_negotiation.cc:407] Connection header received
0929 10:54:39.663932 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:39.663939 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:39.663985 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:39.664135 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:39.664748 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:39.665840 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:39.666641 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:39.667056 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:39.667175 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:39.669578 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:39.669582 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:39.669583 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:39.669619 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:39.671792 (+ 2173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:39.672191 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:39.672194 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:39.672195 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:39.672308 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:39.672562 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:39.672563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:39.672563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:39.672915 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:39.673021 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:39.673294 (+ 273us) server_negotiation.cc:299] Negotiation successful
0929 10:54:39.673439 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":84,"threads_started":1}
I20260929 10:54:39.674726 12373 catalog_manager.cc:1347] Generated new cluster ID: b19ea84a8cab4b5fb197a5caf2dc74b5
I20260929 10:54:39.674777 12373 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:54:39.678434 12373 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:54:39.678928 12373 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:54:39.684100 12373 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c26351720c0f48109cd9a0bb30c62f84: Generated new TSK 0
I20260929 10:54:39.684307 12373 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.238.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab.
Entry for principal kudu/127.10.238.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab.
I20260929 10:54:39.711458 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.193:0
--local_ip_for_outbound_sockets=127.10.238.193
--webserver_interface=127.10.238.193
--webserver_port=0
--tserver_master_addrs=127.10.238.254:41855
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab
--principal=kudu/127.10.238.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:37781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:54:39.794008 12382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:54:39.794235 12382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:54:39.794276 12382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:54:39.795424 12382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:54:39.795483 12382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:54:39.795516 12382 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 10:54:39.795553 12382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.193
I20260929 10:54:39.797029 12382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:37781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab
--principal=kudu/127.10.238.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.238.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.238.254:41855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.193
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:54:39.797339 12382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:54:39.797561 12382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:54:39.802976 12388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:39.802999 12387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:39.802989 12390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:39.803310 12382 server_base.cc:1034] running on GCE node
I20260929 10:54:39.803524 12382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:54:39.803752 12382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:54:39.804915 12382 hybrid_clock.cc:648] HybridClock initialized: now 1790679279804911 us; error 36 us; skew 500 ppm
Sep 29 10:54:39 dist-test-slave-8nrt krb5kdc[12265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679279, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:54:39.807412 12382 init.cc:377] Logged in from keytab as kudu/127.10.238.193@KRBTEST.COM (short username kudu)
I20260929 10:54:39.807986 12382 webserver.cc:460] Webserver started at http://127.10.238.193:38349/ using document root <none> and password file <none>
I20260929 10:54:39.808270 12382 fs_manager.cc:359] Metadata directory not provided
I20260929 10:54:39.808346 12382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:54:39.808501 12382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:54:39.809648 12382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-0/data/instance:
uuid: "91e8b37d2bcc4041a11630f9f303607d"
format_stamp: "Formatted at 2026-09-29 10:54:39 on dist-test-slave-8nrt"
server_key: "cf56b99558daa1c427669ba3e2e537a6"
server_key_iv: "5a731ea0b51bf3977000f0617bed8d4f"
server_key_version: "encryptionkey@0"
I20260929 10:54:39.809952 12382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-0/wal/instance:
uuid: "91e8b37d2bcc4041a11630f9f303607d"
format_stamp: "Formatted at 2026-09-29 10:54:39 on dist-test-slave-8nrt"
server_key: "cf56b99558daa1c427669ba3e2e537a6"
server_key_iv: "5a731ea0b51bf3977000f0617bed8d4f"
server_key_version: "encryptionkey@0"
I20260929 10:54:39.811151 12382 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:54:39.811818 12397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:39.811966 12382 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:54:39.812062 12382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-0/wal
uuid: "91e8b37d2bcc4041a11630f9f303607d"
format_stamp: "Formatted at 2026-09-29 10:54:39 on dist-test-slave-8nrt"
server_key: "cf56b99558daa1c427669ba3e2e537a6"
server_key_iv: "5a731ea0b51bf3977000f0617bed8d4f"
server_key_version: "encryptionkey@0"
I20260929 10:54:39.812114 12382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:54:39.824111 12382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:54:39.825827 12382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:54:39.825928 12382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:54:39.826332 12382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:54:39.826365 12382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:39.826390 12382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:54:39.826431 12382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:39.833446 12382 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.193:40549
Sep 29 10:54:39 dist-test-slave-8nrt krb5kdc[12265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679279, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.193@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:54:39.833786 12382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
I20260929 10:54:39.834059 12512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.193:40549 every 8 connection(s)
I20260929 10:54:39.836364 12359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:39.826789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:52871 (local address 127.10.238.254:41855)
0929 10:54:39.826860 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:39.826863 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:39.827112 (+ 249us) server_negotiation.cc:407] Connection header received
0929 10:54:39.827572 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:39.827573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:39.827587 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:39.827671 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:39.828486 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:39.829329 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:39.830942 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:39.831290 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:39.831370 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:39.833977 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:39.833979 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:39.833980 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:39.833988 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:39.835236 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:39.835592 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:39.835592 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:39.835593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:39.835642 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:39.835855 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:39.835855 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:39.835856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:39.835955 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:39.836004 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:39.836222 (+ 218us) server_negotiation.cc:299] Negotiation successful
0929 10:54:39.836287 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 10:54:39.836402 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 12382
I20260929 10:54:39.836493 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-0/wal/instance
I20260929 10:54:39.836637 12406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:39.826784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:41855 (local address 127.10.238.193:52871)
0929 10:54:39.826966 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:39.827003 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:39.827483 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:39.827752 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:39.827756 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:39.828100 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:54:39.828377 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:39.828384 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:39.829417 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:39.829418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:39.830851 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:39.830855 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:39.831406 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:39.831407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:39.831565 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:39.832061 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:54:39.832071 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:54:39.833876 (+ 1805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:54:39.835313 (+ 1437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:39.835319 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:39.835321 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:39.835525 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:39.835698 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:39.835699 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:39.835699 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:39.835800 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:39.836010 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:54:39.836015 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:54:39.836118 (+ 103us) client_negotiation.cc:769] Sending connection context
0929 10:54:39.836196 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 10:54:39.836396 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":104,"threads_started":1}
I20260929 10:54:39.836658 11195 external_mini_cluster.cc:1428] Setting key e57c93bf72f08bee0d4cb189c8cf1d8c
I20260929 10:54:39.837165 12513 heartbeater.cc:347] Connected to a master server at 127.10.238.254:41855
I20260929 10:54:39.837291 12513 heartbeater.cc:464] Registering TS with master...
I20260929 10:54:39.837474 12513 heartbeater.cc:511] Master 127.10.238.254:41855 requested a full tablet report, sending...
I20260929 10:54:39.838037 12310 ts_manager.cc:194] Registered new tserver with Master: 91e8b37d2bcc4041a11630f9f303607d (127.10.238.193:40549)
I20260929 10:54:39.838792 12310 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.193@KRBTEST.COM'} at 127.10.238.193:52871
I20260929 10:54:39.844736 12519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:39.838332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:41855 (local address 127.10.238.193:60003)
0929 10:54:39.838449 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:39.838459 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:39.838509 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:39.839098 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:39.839099 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:39.839317 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:54:39.839387 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:39.839389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:39.840263 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:39.840264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:39.840747 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:39.840749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:39.841307 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:39.841308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:39.841392 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:39.841826 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:54:39.841832 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:54:39.842100 (+ 268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:54:39.843812 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:39.843813 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:39.843813 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:39.843971 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:39.844187 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:39.844187 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:39.844188 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:39.844236 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:39.844516 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:54:39.844517 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:54:39.844551 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 10:54:39.844599 (+ 48us) client_negotiation.cc:240] Negotiation successful
0929 10:54:39.844668 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":65,"threads_started":1}
I20260929 10:54:39.844789 12359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:39.838329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:60003 (local address 127.10.238.254:41855)
0929 10:54:39.838980 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:39.838981 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:39.838988 (+ 7us) server_negotiation.cc:407] Connection header received
0929 10:54:39.839002 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:39.839003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:39.839027 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:39.839085 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:39.839472 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:39.840158 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:39.840830 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:39.841161 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:39.841282 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:39.842183 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:39.842185 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:39.842185 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:39.842192 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:39.843728 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:39.844043 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:39.844044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:39.844045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:39.844107 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:39.844312 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:39.844313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:39.844313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:39.844449 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:39.844524 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:39.844624 (+ 100us) server_negotiation.cc:299] Negotiation successful
0929 10:54:39.844703 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636}
WARNING: no policy specified for kudu/127.10.238.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab.
Entry for principal kudu/127.10.238.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab.
I20260929 10:54:39.862172 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.194:0
--local_ip_for_outbound_sockets=127.10.238.194
--webserver_interface=127.10.238.194
--webserver_port=0
--tserver_master_addrs=127.10.238.254:41855
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab
--principal=kudu/127.10.238.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:37781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:54:39.942999 12522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:54:39.943207 12522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:54:39.943233 12522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:54:39.944365 12522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:54:39.944404 12522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:54:39.944427 12522 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 10:54:39.944445 12522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.194
I20260929 10:54:39.945861 12522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:37781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab
--principal=kudu/127.10.238.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.238.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.238.254:41855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.194
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:54:39.946112 12522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:54:39.946317 12522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:54:39.951359 12528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:39.951382 12527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:39.951382 12530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:39.951442 12522 server_base.cc:1034] running on GCE node
I20260929 10:54:39.951731 12522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:54:39.951922 12522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:54:39.953071 12522 hybrid_clock.cc:648] HybridClock initialized: now 1790679279953061 us; error 29 us; skew 500 ppm
Sep 29 10:54:39 dist-test-slave-8nrt krb5kdc[12265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679279, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:54:39.955924 12522 init.cc:377] Logged in from keytab as kudu/127.10.238.194@KRBTEST.COM (short username kudu)
I20260929 10:54:39.956369 12522 webserver.cc:460] Webserver started at http://127.10.238.194:38011/ using document root <none> and password file <none>
I20260929 10:54:39.956629 12522 fs_manager.cc:359] Metadata directory not provided
I20260929 10:54:39.956696 12522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:54:39.956833 12522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:54:39.957906 12522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-1/data/instance:
uuid: "8f22daec740b421ab09c01da42631470"
format_stamp: "Formatted at 2026-09-29 10:54:39 on dist-test-slave-8nrt"
server_key: "9b60cde8295b8841a315827bd5bf1c72"
server_key_iv: "195fc0d425571558ae26ed3a363a9709"
server_key_version: "encryptionkey@0"
I20260929 10:54:39.958253 12522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-1/wal/instance:
uuid: "8f22daec740b421ab09c01da42631470"
format_stamp: "Formatted at 2026-09-29 10:54:39 on dist-test-slave-8nrt"
server_key: "9b60cde8295b8841a315827bd5bf1c72"
server_key_iv: "195fc0d425571558ae26ed3a363a9709"
server_key_version: "encryptionkey@0"
I20260929 10:54:39.959537 12522 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 10:54:39.960270 12537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:39.960462 12522 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:54:39.960556 12522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-1/wal
uuid: "8f22daec740b421ab09c01da42631470"
format_stamp: "Formatted at 2026-09-29 10:54:39 on dist-test-slave-8nrt"
server_key: "9b60cde8295b8841a315827bd5bf1c72"
server_key_iv: "195fc0d425571558ae26ed3a363a9709"
server_key_version: "encryptionkey@0"
I20260929 10:54:39.960632 12522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:54:39.973955 12522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:54:39.975728 12522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:54:39.975886 12522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:54:39.976478 12522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:54:39.976529 12522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:39.976560 12522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:54:39.976578 12522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 10:54:39 dist-test-slave-8nrt krb5kdc[12265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679279, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.194@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:54:39.984038 12522 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.194:40521
I20260929 10:54:39.984145 12652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.194:40521 every 8 connection(s)
I20260929 10:54:39.984433 12522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
I20260929 10:54:39.986534 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 12522
I20260929 10:54:39.986625 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-1/wal/instance
I20260929 10:54:39.986686 12359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:39.977112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:33451 (local address 127.10.238.254:41855)
0929 10:54:39.977169 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:39.977171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:39.977181 (+ 10us) server_negotiation.cc:407] Connection header received
0929 10:54:39.977874 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:39.977875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:39.977887 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:39.977974 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:39.978917 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:39.979444 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:39.981048 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:39.981246 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:39.981356 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:39.983739 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:39.983740 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:39.983741 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:39.983751 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:39.985450 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:39.985860 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:39.985861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:39.985861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:39.985911 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:39.986103 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:39.986104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:39.986104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:39.986198 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:39.986244 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:39.986499 (+ 255us) server_negotiation.cc:299] Negotiation successful
0929 10:54:39.986577 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 10:54:39.986814 11195 external_mini_cluster.cc:1428] Setting key b14ae7c20371a26b893fa851ff953658
I20260929 10:54:39.986825 12546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:39.976837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:41855 (local address 127.10.238.194:33451)
0929 10:54:39.977012 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:39.977064 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:39.977754 (+ 690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:39.978060 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:39.978065 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:39.978474 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:54:39.978784 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:39.978791 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:39.979514 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:39.979515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:39.980943 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:39.980946 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:39.981382 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:39.981383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:39.981513 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:39.982032 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:54:39.982040 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:54:39.983635 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:54:39.985516 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:39.985520 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:39.985522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:39.985800 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:39.985963 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:39.985964 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:39.985964 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:39.986055 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:39.986248 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:54:39.986253 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:54:39.986352 (+ 99us) client_negotiation.cc:769] Sending connection context
0929 10:54:39.986446 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 10:54:39.986554 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":78,"threads_started":1}
I20260929 10:54:39.987268 12653 heartbeater.cc:347] Connected to a master server at 127.10.238.254:41855
I20260929 10:54:39.987350 12653 heartbeater.cc:464] Registering TS with master...
I20260929 10:54:39.987533 12653 heartbeater.cc:511] Master 127.10.238.254:41855 requested a full tablet report, sending...
I20260929 10:54:39.987890 12308 ts_manager.cc:194] Registered new tserver with Master: 8f22daec740b421ab09c01da42631470 (127.10.238.194:40521)
I20260929 10:54:39.988337 12308 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.194@KRBTEST.COM'} at 127.10.238.194:33451
I20260929 10:54:39.994644 12359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:39.988301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:43019 (local address 127.10.238.254:41855)
0929 10:54:39.988337 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:39.988339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:39.988767 (+ 428us) server_negotiation.cc:407] Connection header received
0929 10:54:39.988794 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:39.988795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:39.988805 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:39.988862 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:39.989248 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:39.989736 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:39.990689 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:39.990879 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:39.990932 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:39.991901 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:39.991902 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:39.991902 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:39.991908 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:39.993358 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:39.993799 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:39.993800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:39.993800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:39.993859 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:39.994099 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:39.994100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:39.994100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:39.994312 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:39.994376 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:39.994534 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 10:54:39.994588 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 10:54:39.994740 12659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:39.988369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:41855 (local address 127.10.238.194:43019)
0929 10:54:39.988680 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:39.988691 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:39.988751 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:39.988869 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:39.988869 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:39.989057 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:54:39.989127 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:39.989129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:39.989838 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:39.989839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:39.990615 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:39.990618 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:39.990950 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:39.990951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:39.991029 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:39.991381 (+ 352us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:54:39.991387 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:54:39.991799 (+ 412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:54:39.993428 (+ 1629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:39.993429 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:39.993429 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:39.993702 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:39.993943 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:39.993944 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:39.993944 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:39.994012 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:39.994374 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:54:39.994375 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:54:39.994423 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 10:54:39.994519 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 10:54:39.994617 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":108,"threads_started":1}
WARNING: no policy specified for kudu/127.10.238.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab.
Entry for principal kudu/127.10.238.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab.
I20260929 10:54:40.012301 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.195:0
--local_ip_for_outbound_sockets=127.10.238.195
--webserver_interface=127.10.238.195
--webserver_port=0
--tserver_master_addrs=127.10.238.254:41855
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab
--principal=kudu/127.10.238.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:37781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:54:40.091950 12662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:54:40.092132 12662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:54:40.092154 12662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:54:40.093190 12662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:54:40.093221 12662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:54:40.093241 12662 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 10:54:40.093283 12662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.195
I20260929 10:54:40.094570 12662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:37781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab
--principal=kudu/127.10.238.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.238.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.238.254:41855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.195
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:54:40.094758 12662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:54:40.094930 12662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:54:40.099764 12670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:40.099764 12667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:40.099958 12662 server_base.cc:1034] running on GCE node
W20260929 10:54:40.099777 12668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:40.100243 12662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:54:40.100479 12662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:54:40.101641 12662 hybrid_clock.cc:648] HybridClock initialized: now 1790679280101626 us; error 47 us; skew 500 ppm
Sep 29 10:54:40 dist-test-slave-8nrt krb5kdc[12265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679280, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:54:40.104238 12662 init.cc:377] Logged in from keytab as kudu/127.10.238.195@KRBTEST.COM (short username kudu)
I20260929 10:54:40.104630 12662 webserver.cc:460] Webserver started at http://127.10.238.195:35095/ using document root <none> and password file <none>
I20260929 10:54:40.104885 12662 fs_manager.cc:359] Metadata directory not provided
I20260929 10:54:40.104955 12662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:54:40.105103 12662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:54:40.106140 12662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-2/data/instance:
uuid: "9bcd53f3752d4d80bcfcbba0c194d9a9"
format_stamp: "Formatted at 2026-09-29 10:54:40 on dist-test-slave-8nrt"
server_key: "a9b17da834f60aaed9f24fd771b492c3"
server_key_iv: "79ffcb43618fe9dcfbcfd86e16fa9a9a"
server_key_version: "encryptionkey@0"
I20260929 10:54:40.106484 12662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-2/wal/instance:
uuid: "9bcd53f3752d4d80bcfcbba0c194d9a9"
format_stamp: "Formatted at 2026-09-29 10:54:40 on dist-test-slave-8nrt"
server_key: "a9b17da834f60aaed9f24fd771b492c3"
server_key_iv: "79ffcb43618fe9dcfbcfd86e16fa9a9a"
server_key_version: "encryptionkey@0"
I20260929 10:54:40.107751 12662 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 10:54:40.108445 12677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:40.108629 12662 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:54:40.108750 12662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-2/wal
uuid: "9bcd53f3752d4d80bcfcbba0c194d9a9"
format_stamp: "Formatted at 2026-09-29 10:54:40 on dist-test-slave-8nrt"
server_key: "a9b17da834f60aaed9f24fd771b492c3"
server_key_iv: "79ffcb43618fe9dcfbcfd86e16fa9a9a"
server_key_version: "encryptionkey@0"
I20260929 10:54:40.108834 12662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:54:40.131652 12662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:54:40.133977 12662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:54:40.134148 12662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:54:40.134673 12662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:54:40.134748 12662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:40.134791 12662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:54:40.134819 12662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 10:54:40 dist-test-slave-8nrt krb5kdc[12265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679280, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.195@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:54:40.142026 12662 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.195:39331
I20260929 10:54:40.142112 12792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.195:39331 every 8 connection(s)
I20260929 10:54:40.142511 12662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
I20260929 10:54:40.144732 12359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:40.135165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:48249 (local address 127.10.238.254:41855)
0929 10:54:40.135234 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:40.135237 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:40.135446 (+ 209us) server_negotiation.cc:407] Connection header received
0929 10:54:40.135933 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:40.135935 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:40.135949 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:40.136027 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:40.136892 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.137756 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.138894 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.139105 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.139191 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:40.141769 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:40.141771 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:40.141771 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:40.141780 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:40.143547 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:40.143933 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:40.143934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:40.143934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:40.143991 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:40.144214 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:40.144214 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:40.144215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:40.144320 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:40.144372 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:40.144588 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 10:54:40.144661 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 10:54:40.144892 12686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:40.135155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:41855 (local address 127.10.238.195:48249)
0929 10:54:40.135310 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:40.135349 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:40.135835 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:40.136131 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:40.136135 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:40.136500 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:54:40.136786 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:40.136793 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.137855 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.137856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:40.138804 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:40.138807 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.139217 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.139218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:40.139346 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:40.139834 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:54:40.139851 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:54:40.141655 (+ 1804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:54:40.143613 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:40.143618 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:40.143620 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:40.143840 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:40.144061 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:40.144062 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:40.144062 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:40.144157 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:40.144374 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:54:40.144376 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:54:40.144481 (+ 105us) client_negotiation.cc:769] Sending connection context
0929 10:54:40.144558 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 10:54:40.144672 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
I20260929 10:54:40.145442 12793 heartbeater.cc:347] Connected to a master server at 127.10.238.254:41855
I20260929 10:54:40.145524 12793 heartbeater.cc:464] Registering TS with master...
I20260929 10:54:40.145709 12793 heartbeater.cc:511] Master 127.10.238.254:41855 requested a full tablet report, sending...
I20260929 10:54:40.146076 12310 ts_manager.cc:194] Registered new tserver with Master: 9bcd53f3752d4d80bcfcbba0c194d9a9 (127.10.238.195:39331)
I20260929 10:54:40.146453 12310 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.195@KRBTEST.COM'} at 127.10.238.195:48249
I20260929 10:54:40.146793 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 12662
I20260929 10:54:40.146888 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-2/wal/instance
I20260929 10:54:40.147068 11195 external_mini_cluster.cc:1428] Setting key 839b57821edc2084f3d865fd5b9eb8e9
I20260929 10:54:40.147912 11195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:54:40.152665 12359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:40.146523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:58541 (local address 127.10.238.254:41855)
0929 10:54:40.146565 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:40.146567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:40.146794 (+ 227us) server_negotiation.cc:407] Connection header received
0929 10:54:40.146852 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:40.146853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:40.146867 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:40.146936 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:40.147284 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.147985 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.148681 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.149003 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.149076 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:40.150110 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:40.150111 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:40.150111 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:40.150117 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:40.151504 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:40.151906 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:40.151907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:40.151908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:40.151969 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:40.152169 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:40.152170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:40.152170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:40.152280 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:40.152354 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:40.152506 (+ 152us) server_negotiation.cc:299] Negotiation successful
0929 10:54:40.152578 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 10:54:40.152683 12799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:40.146481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:41855 (local address 127.10.238.195:58541)
0929 10:54:40.146724 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:40.146734 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:40.146802 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:40.146943 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:40.146943 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:40.147134 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:54:40.147188 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:40.147190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.148116 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.148116 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:40.148603 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:40.148604 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.149079 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.149079 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:40.149134 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:40.149599 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:54:40.149606 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:54:40.150014 (+ 408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:54:40.151655 (+ 1641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:40.151656 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:40.151656 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:40.151820 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:40.152033 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:40.152034 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:40.152034 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:40.152096 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:40.152346 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:54:40.152347 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:54:40.152394 (+ 47us) client_negotiation.cc:769] Sending connection context
0929 10:54:40.152480 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 10:54:40.152579 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":192,"threads_started":1}
Sep 29 10:54:40 dist-test-slave-8nrt krb5kdc[12265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679280, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 10:54:40 dist-test-slave-8nrt krb5kdc[12265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679280, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:54:40.171461 12359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:40.164550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38618 (local address 127.10.238.254:41855)
0929 10:54:40.164593 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:40.164595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:40.164685 (+ 90us) server_negotiation.cc:407] Connection header received
0929 10:54:40.164729 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:40.164730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:40.164739 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:40.164782 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:40.165343 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.165813 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.166566 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.166737 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.166787 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:40.168931 (+ 2144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:40.168933 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:40.168933 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:40.168939 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:40.170340 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:40.170711 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:40.170712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:40.170713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:40.170764 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:40.170982 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:40.170983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:40.170983 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:40.171085 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:40.171162 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:40.171266 (+ 104us) server_negotiation.cc:299] Negotiation successful
0929 10:54:40.171350 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 10:54:40.173246 12310 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38618:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:54:40.174923 12310 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:54:40.181241 12812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:40.177442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51546 (local address 127.10.238.194:40521)
0929 10:54:40.177667 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:40.177672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:40.177688 (+ 16us) server_negotiation.cc:407] Connection header received
0929 10:54:40.177719 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:40.177720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:40.177732 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:40.177815 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:54:40.178032 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.178816 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.180140 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.180811 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.180900 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:40.180957 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:40.181106 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 10:54:40.181147 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":87,"threads_started":1}
I20260929 10:54:40.181231 12811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:40.177351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.194:40521 (local address 127.0.0.1:51546)
0929 10:54:40.177477 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:40.177496 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:40.177543 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:40.177825 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:40.177826 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:40.177828 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:54:40.177876 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:40.177878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.178910 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.178911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:40.180042 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:40.180044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.180895 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.180896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:40.180972 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:40.180986 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 10:54:40.181085 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 10:54:40.181146 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":49,"threads_started":1}
I20260929 10:54:40.181695 12809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:40.177224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36640 (local address 127.10.238.193:40549)
0929 10:54:40.177433 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:40.177439 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:40.177543 (+ 104us) server_negotiation.cc:407] Connection header received
0929 10:54:40.177672 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:40.177674 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:40.177695 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:40.177791 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:54:40.177952 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.179092 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.180392 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.181134 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.181209 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:40.181290 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:40.181536 (+ 246us) server_negotiation.cc:299] Negotiation successful
0929 10:54:40.181581 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":97,"threads_started":1}
I20260929 10:54:40.181774 12807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:40.177098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.195:39331 (local address 127.0.0.1:55476)
0929 10:54:40.177448 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:40.177479 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:40.177543 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:40.178172 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:40.178173 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:40.178175 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:54:40.178226 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:40.178227 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.179412 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.179413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:40.180477 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:40.180479 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.181509 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.181510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:40.181581 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:40.181584 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 10:54:40.181644 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 10:54:40.181687 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":2,"thread_start_us":71,"threads_started":1}
I20260929 10:54:40.181839 12810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:40.177210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55476 (local address 127.10.238.195:39331)
0929 10:54:40.177961 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:40.177965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:40.177975 (+ 10us) server_negotiation.cc:407] Connection header received
0929 10:54:40.178013 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:40.178015 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:40.178030 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:40.178103 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:54:40.178315 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.179327 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.180557 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.181319 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.181385 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:40.181473 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:40.181693 (+ 220us) server_negotiation.cc:299] Negotiation successful
0929 10:54:40.181740 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":712,"thread_start_us":119,"threads_started":1}
I20260929 10:54:40.182013 12808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:40.177098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:40549 (local address 127.0.0.1:36640)
0929 10:54:40.177428 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:40.177473 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:40.177571 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:40.177792 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:40.177794 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:40.177797 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:54:40.177863 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:40.177865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.179192 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.179193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:40.180300 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:40.180302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.181329 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.181331 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:40.181418 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:40.181422 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 10:54:40.181857 (+ 435us) client_negotiation.cc:240] Negotiation successful
0929 10:54:40.181952 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":65,"threads_started":1}
I20260929 10:54:40.182441 12587 tablet_service.cc:1467] Processing CreateTablet for tablet 6e67e8c44f764aba8de65f51823c3866 (DEFAULT_TABLE table=test-table [id=e04ebb5123324c7f9a0391fa57ae0e1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:54:40.182552 12727 tablet_service.cc:1467] Processing CreateTablet for tablet 6e67e8c44f764aba8de65f51823c3866 (DEFAULT_TABLE table=test-table [id=e04ebb5123324c7f9a0391fa57ae0e1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:54:40.182654 12447 tablet_service.cc:1467] Processing CreateTablet for tablet 6e67e8c44f764aba8de65f51823c3866 (DEFAULT_TABLE table=test-table [id=e04ebb5123324c7f9a0391fa57ae0e1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:54:40.182835 12587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e67e8c44f764aba8de65f51823c3866. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:54:40.182904 12727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e67e8c44f764aba8de65f51823c3866. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:54:40.182940 12447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e67e8c44f764aba8de65f51823c3866. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:54:40.185747 12813 tablet_bootstrap.cc:492] T 6e67e8c44f764aba8de65f51823c3866 P 9bcd53f3752d4d80bcfcbba0c194d9a9: Bootstrap starting.
I20260929 10:54:40.186484 12814 tablet_bootstrap.cc:492] T 6e67e8c44f764aba8de65f51823c3866 P 8f22daec740b421ab09c01da42631470: Bootstrap starting.
I20260929 10:54:40.186702 12813 tablet_bootstrap.cc:654] T 6e67e8c44f764aba8de65f51823c3866 P 9bcd53f3752d4d80bcfcbba0c194d9a9: Neither blocks nor log segments found. Creating new log.
I20260929 10:54:40.186734 12815 tablet_bootstrap.cc:492] T 6e67e8c44f764aba8de65f51823c3866 P 91e8b37d2bcc4041a11630f9f303607d: Bootstrap starting.
I20260929 10:54:40.187064 12813 log.cc:824] T 6e67e8c44f764aba8de65f51823c3866 P 9bcd53f3752d4d80bcfcbba0c194d9a9: Log is configured to *not* fsync() on all Append() calls
I20260929 10:54:40.187386 12815 tablet_bootstrap.cc:654] T 6e67e8c44f764aba8de65f51823c3866 P 91e8b37d2bcc4041a11630f9f303607d: Neither blocks nor log segments found. Creating new log.
I20260929 10:54:40.187752 12815 log.cc:824] T 6e67e8c44f764aba8de65f51823c3866 P 91e8b37d2bcc4041a11630f9f303607d: Log is configured to *not* fsync() on all Append() calls
I20260929 10:54:40.187876 12813 tablet_bootstrap.cc:492] T 6e67e8c44f764aba8de65f51823c3866 P 9bcd53f3752d4d80bcfcbba0c194d9a9: No bootstrap required, opened a new log
I20260929 10:54:40.187948 12813 ts_tablet_manager.cc:1397] T 6e67e8c44f764aba8de65f51823c3866 P 9bcd53f3752d4d80bcfcbba0c194d9a9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 10:54:40.188490 12815 tablet_bootstrap.cc:492] T 6e67e8c44f764aba8de65f51823c3866 P 91e8b37d2bcc4041a11630f9f303607d: No bootstrap required, opened a new log
I20260929 10:54:40.188556 12815 ts_tablet_manager.cc:1397] T 6e67e8c44f764aba8de65f51823c3866 P 91e8b37d2bcc4041a11630f9f303607d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 10:54:40.188568 12814 tablet_bootstrap.cc:654] T 6e67e8c44f764aba8de65f51823c3866 P 8f22daec740b421ab09c01da42631470: Neither blocks nor log segments found. Creating new log.
I20260929 10:54:40.188906 12814 log.cc:824] T 6e67e8c44f764aba8de65f51823c3866 P 8f22daec740b421ab09c01da42631470: Log is configured to *not* fsync() on all Append() calls
I20260929 10:54:40.189704 12814 tablet_bootstrap.cc:492] T 6e67e8c44f764aba8de65f51823c3866 P 8f22daec740b421ab09c01da42631470: No bootstrap required, opened a new log
I20260929 10:54:40.189772 12814 ts_tablet_manager.cc:1397] T 6e67e8c44f764aba8de65f51823c3866 P 8f22daec740b421ab09c01da42631470: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 10:54:40.190210 12813 raft_consensus.cc:348] T 6e67e8c44f764aba8de65f51823c3866 P 9bcd53f3752d4d80bcfcbba0c194d9a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e8b37d2bcc4041a11630f9f303607d" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 40549 } } peers { permanent_uuid: "9bcd53f3752d4d80bcfcbba0c194d9a9" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 39331 } } peers { permanent_uuid: "8f22daec740b421ab09c01da42631470" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40521 } }
I20260929 10:54:40.190383 12813 raft_consensus.cc:374] T 6e67e8c44f764aba8de65f51823c3866 P 9bcd53f3752d4d80bcfcbba0c194d9a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:54:40.190284 12815 raft_consensus.cc:348] T 6e67e8c44f764aba8de65f51823c3866 P 91e8b37d2bcc4041a11630f9f303607d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e8b37d2bcc4041a11630f9f303607d" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 40549 } } peers { permanent_uuid: "9bcd53f3752d4d80bcfcbba0c194d9a9" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 39331 } } peers { permanent_uuid: "8f22daec740b421ab09c01da42631470" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40521 } }
I20260929 10:54:40.190418 12813 raft_consensus.cc:729] T 6e67e8c44f764aba8de65f51823c3866 P 9bcd53f3752d4d80bcfcbba0c194d9a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bcd53f3752d4d80bcfcbba0c194d9a9, State: Initialized, Role: FOLLOWER
I20260929 10:54:40.190425 12815 raft_consensus.cc:374] T 6e67e8c44f764aba8de65f51823c3866 P 91e8b37d2bcc4041a11630f9f303607d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:54:40.190459 12815 raft_consensus.cc:729] T 6e67e8c44f764aba8de65f51823c3866 P 91e8b37d2bcc4041a11630f9f303607d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91e8b37d2bcc4041a11630f9f303607d, State: Initialized, Role: FOLLOWER
I20260929 10:54:40.190538 12813 consensus_queue.cc:260] T 6e67e8c44f764aba8de65f51823c3866 P 9bcd53f3752d4d80bcfcbba0c194d9a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e8b37d2bcc4041a11630f9f303607d" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 40549 } } peers { permanent_uuid: "9bcd53f3752d4d80bcfcbba0c194d9a9" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 39331 } } peers { permanent_uuid: "8f22daec740b421ab09c01da42631470" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40521 } }
I20260929 10:54:40.190577 12815 consensus_queue.cc:260] T 6e67e8c44f764aba8de65f51823c3866 P 91e8b37d2bcc4041a11630f9f303607d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e8b37d2bcc4041a11630f9f303607d" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 40549 } } peers { permanent_uuid: "9bcd53f3752d4d80bcfcbba0c194d9a9" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 39331 } } peers { permanent_uuid: "8f22daec740b421ab09c01da42631470" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40521 } }
I20260929 10:54:40.190793 12813 ts_tablet_manager.cc:1428] T 6e67e8c44f764aba8de65f51823c3866 P 9bcd53f3752d4d80bcfcbba0c194d9a9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 10:54:40.190848 12815 ts_tablet_manager.cc:1428] T 6e67e8c44f764aba8de65f51823c3866 P 91e8b37d2bcc4041a11630f9f303607d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 10:54:40.190923 12793 heartbeater.cc:503] Master 127.10.238.254:41855 was elected leader, sending a full tablet report...
I20260929 10:54:40.191316 12513 heartbeater.cc:503] Master 127.10.238.254:41855 was elected leader, sending a full tablet report...
I20260929 10:54:40.191854 12814 raft_consensus.cc:348] T 6e67e8c44f764aba8de65f51823c3866 P 8f22daec740b421ab09c01da42631470 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e8b37d2bcc4041a11630f9f303607d" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 40549 } } peers { permanent_uuid: "9bcd53f3752d4d80bcfcbba0c194d9a9" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 39331 } } peers { permanent_uuid: "8f22daec740b421ab09c01da42631470" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40521 } }
I20260929 10:54:40.192025 12814 raft_consensus.cc:374] T 6e67e8c44f764aba8de65f51823c3866 P 8f22daec740b421ab09c01da42631470 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:54:40.192072 12814 raft_consensus.cc:729] T 6e67e8c44f764aba8de65f51823c3866 P 8f22daec740b421ab09c01da42631470 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f22daec740b421ab09c01da42631470, State: Initialized, Role: FOLLOWER
I20260929 10:54:40.192178 12814 consensus_queue.cc:260] T 6e67e8c44f764aba8de65f51823c3866 P 8f22daec740b421ab09c01da42631470 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e8b37d2bcc4041a11630f9f303607d" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 40549 } } peers { permanent_uuid: "9bcd53f3752d4d80bcfcbba0c194d9a9" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 39331 } } peers { permanent_uuid: "8f22daec740b421ab09c01da42631470" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40521 } }
I20260929 10:54:40.192433 12814 ts_tablet_manager.cc:1428] T 6e67e8c44f764aba8de65f51823c3866 P 8f22daec740b421ab09c01da42631470: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 10:54:40.192520 12653 heartbeater.cc:503] Master 127.10.238.254:41855 was elected leader, sending a full tablet report...
W20260929 10:54:40.235025 12654 tablet.cc:2369] T 6e67e8c44f764aba8de65f51823c3866 P 8f22daec740b421ab09c01da42631470: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:54:40.310678 12820 raft_consensus.cc:482] T 6e67e8c44f764aba8de65f51823c3866 P 91e8b37d2bcc4041a11630f9f303607d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:54:40.310810 12820 raft_consensus.cc:504] T 6e67e8c44f764aba8de65f51823c3866 P 91e8b37d2bcc4041a11630f9f303607d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e8b37d2bcc4041a11630f9f303607d" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 40549 } } peers { permanent_uuid: "9bcd53f3752d4d80bcfcbba0c194d9a9" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 39331 } } peers { permanent_uuid: "8f22daec740b421ab09c01da42631470" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40521 } }
I20260929 10:54:40.311201 12820 leader_election.cc:288] T 6e67e8c44f764aba8de65f51823c3866 P 91e8b37d2bcc4041a11630f9f303607d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bcd53f3752d4d80bcfcbba0c194d9a9 (127.10.238.195:39331), 8f22daec740b421ab09c01da42631470 (127.10.238.194:40521)
I20260929 10:54:40.314234 12810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:40.311659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:58051 (local address 127.10.238.195:39331)
0929 10:54:40.311710 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:40.311712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:40.311722 (+ 10us) server_negotiation.cc:407] Connection header received
0929 10:54:40.311752 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:40.311754 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:40.311778 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:40.311861 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:54:40.312017 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.312597 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.313550 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.313889 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.313968 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:40.313977 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:40.314116 (+ 139us) server_negotiation.cc:299] Negotiation successful
0929 10:54:40.314150 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 10:54:40.314234 12406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:40.311438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.195:39331 (local address 127.10.238.193:58051)
0929 10:54:40.311496 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:40.311508 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:40.311555 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:40.311870 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:40.311871 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:40.311873 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:54:40.311945 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:40.311946 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.312681 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.312682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:40.313464 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:40.313466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.313967 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.313968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:40.314046 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:40.314049 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 10:54:40.314109 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 10:54:40.314149 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 10:54:40.314541 12747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e67e8c44f764aba8de65f51823c3866" candidate_uuid: "91e8b37d2bcc4041a11630f9f303607d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bcd53f3752d4d80bcfcbba0c194d9a9" is_pre_election: true
I20260929 10:54:40.314620 12812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:40.311664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:50937 (local address 127.10.238.194:40521)
0929 10:54:40.311729 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:40.311731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:40.311859 (+ 128us) server_negotiation.cc:407] Connection header received
0929 10:54:40.311934 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:40.311935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:40.311944 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:40.312012 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:54:40.312176 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.312846 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.313776 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.314120 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.314229 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:40.314238 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:40.314512 (+ 274us) server_negotiation.cc:299] Negotiation successful
0929 10:54:40.314552 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260929 10:54:40.314628 12822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:40.311567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.194:40521 (local address 127.10.238.193:50937)
0929 10:54:40.311773 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:40.311783 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:40.311869 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:40.312014 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:40.312014 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:40.312016 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:54:40.312107 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:40.312109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.312937 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.312938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:40.313690 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:40.313692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.314255 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.314256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:40.314396 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:40.314400 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 10:54:40.314502 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 10:54:40.314544 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":119,"threads_started":1}
I20260929 10:54:40.314747 12747 raft_consensus.cc:2454] T 6e67e8c44f764aba8de65f51823c3866 P 9bcd53f3752d4d80bcfcbba0c194d9a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91e8b37d2bcc4041a11630f9f303607d in term 0.
I20260929 10:54:40.315092 12401 leader_election.cc:302] T 6e67e8c44f764aba8de65f51823c3866 P 91e8b37d2bcc4041a11630f9f303607d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91e8b37d2bcc4041a11630f9f303607d, 9bcd53f3752d4d80bcfcbba0c194d9a9; no voters:
I20260929 10:54:40.315217 12820 raft_consensus.cc:2781] T 6e67e8c44f764aba8de65f51823c3866 P 91e8b37d2bcc4041a11630f9f303607d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:54:40.315277 12820 raft_consensus.cc:482] T 6e67e8c44f764aba8de65f51823c3866 P 91e8b37d2bcc4041a11630f9f303607d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:54:40.315303 12820 raft_consensus.cc:3037] T 6e67e8c44f764aba8de65f51823c3866 P 91e8b37d2bcc4041a11630f9f303607d [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:54:40.315454 12607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e67e8c44f764aba8de65f51823c3866" candidate_uuid: "91e8b37d2bcc4041a11630f9f303607d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f22daec740b421ab09c01da42631470" is_pre_election: true
I20260929 10:54:40.315635 12607 raft_consensus.cc:2454] T 6e67e8c44f764aba8de65f51823c3866 P 8f22daec740b421ab09c01da42631470 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91e8b37d2bcc4041a11630f9f303607d in term 0.
I20260929 10:54:40.316272 12820 raft_consensus.cc:504] T 6e67e8c44f764aba8de65f51823c3866 P 91e8b37d2bcc4041a11630f9f303607d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e8b37d2bcc4041a11630f9f303607d" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 40549 } } peers { permanent_uuid: "9bcd53f3752d4d80bcfcbba0c194d9a9" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 39331 } } peers { permanent_uuid: "8f22daec740b421ab09c01da42631470" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40521 } }
I20260929 10:54:40.316434 12820 leader_election.cc:288] T 6e67e8c44f764aba8de65f51823c3866 P 91e8b37d2bcc4041a11630f9f303607d [CANDIDATE]: Term 1 election: Requested vote from peers 9bcd53f3752d4d80bcfcbba0c194d9a9 (127.10.238.195:39331), 8f22daec740b421ab09c01da42631470 (127.10.238.194:40521)
I20260929 10:54:40.316581 12747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e67e8c44f764aba8de65f51823c3866" candidate_uuid: "91e8b37d2bcc4041a11630f9f303607d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bcd53f3752d4d80bcfcbba0c194d9a9"
I20260929 10:54:40.316625 12607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e67e8c44f764aba8de65f51823c3866" candidate_uuid: "91e8b37d2bcc4041a11630f9f303607d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f22daec740b421ab09c01da42631470"
I20260929 10:54:40.316653 12747 raft_consensus.cc:3037] T 6e67e8c44f764aba8de65f51823c3866 P 9bcd53f3752d4d80bcfcbba0c194d9a9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:54:40.316722 12607 raft_consensus.cc:3037] T 6e67e8c44f764aba8de65f51823c3866 P 8f22daec740b421ab09c01da42631470 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:54:40.317473 12747 raft_consensus.cc:2454] T 6e67e8c44f764aba8de65f51823c3866 P 9bcd53f3752d4d80bcfcbba0c194d9a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91e8b37d2bcc4041a11630f9f303607d in term 1.
I20260929 10:54:40.317596 12607 raft_consensus.cc:2454] T 6e67e8c44f764aba8de65f51823c3866 P 8f22daec740b421ab09c01da42631470 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91e8b37d2bcc4041a11630f9f303607d in term 1.
I20260929 10:54:40.317699 12401 leader_election.cc:302] T 6e67e8c44f764aba8de65f51823c3866 P 91e8b37d2bcc4041a11630f9f303607d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91e8b37d2bcc4041a11630f9f303607d, 9bcd53f3752d4d80bcfcbba0c194d9a9; no voters:
I20260929 10:54:40.317799 12820 raft_consensus.cc:2781] T 6e67e8c44f764aba8de65f51823c3866 P 91e8b37d2bcc4041a11630f9f303607d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:54:40.317952 12820 raft_consensus.cc:686] T 6e67e8c44f764aba8de65f51823c3866 P 91e8b37d2bcc4041a11630f9f303607d [term 1 LEADER]: Becoming Leader. State: Replica: 91e8b37d2bcc4041a11630f9f303607d, State: Running, Role: LEADER
I20260929 10:54:40.318059 12820 consensus_queue.cc:237] T 6e67e8c44f764aba8de65f51823c3866 P 91e8b37d2bcc4041a11630f9f303607d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e8b37d2bcc4041a11630f9f303607d" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 40549 } } peers { permanent_uuid: "9bcd53f3752d4d80bcfcbba0c194d9a9" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 39331 } } peers { permanent_uuid: "8f22daec740b421ab09c01da42631470" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40521 } }
I20260929 10:54:40.318969 12308 catalog_manager.cc:5495] T 6e67e8c44f764aba8de65f51823c3866 P 91e8b37d2bcc4041a11630f9f303607d reported cstate change: term changed from 0 to 1, leader changed from <none> to 91e8b37d2bcc4041a11630f9f303607d (127.10.238.193). New cstate: current_term: 1 leader_uuid: "91e8b37d2bcc4041a11630f9f303607d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e8b37d2bcc4041a11630f9f303607d" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 40549 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bcd53f3752d4d80bcfcbba0c194d9a9" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 39331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f22daec740b421ab09c01da42631470" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40521 } health_report { overall_health: UNKNOWN } } }
I20260929 10:54:40.328397 12809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:40.326258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36652 (local address 127.10.238.193:40549)
0929 10:54:40.326308 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:40.326309 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:40.326321 (+ 12us) server_negotiation.cc:407] Connection header received
0929 10:54:40.326414 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:40.326415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:40.326427 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:40.326481 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:54:40.326590 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.326995 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.327754 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.327926 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.327975 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:40.328152 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:54:40.328234 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:54:40.328296 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:40.328335 (+ 39us) server_negotiation.cc:299] Negotiation successful
0929 10:54:40.328351 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260929 10:54:40.330497 12747 raft_consensus.cc:1261] T 6e67e8c44f764aba8de65f51823c3866 P 9bcd53f3752d4d80bcfcbba0c194d9a9 [term 1 FOLLOWER]: Refusing update from remote peer 91e8b37d2bcc4041a11630f9f303607d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:54:40.330497 12607 raft_consensus.cc:1261] T 6e67e8c44f764aba8de65f51823c3866 P 8f22daec740b421ab09c01da42631470 [term 1 FOLLOWER]: Refusing update from remote peer 91e8b37d2bcc4041a11630f9f303607d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:54:40.330834 12823 consensus_queue.cc:1035] T 6e67e8c44f764aba8de65f51823c3866 P 91e8b37d2bcc4041a11630f9f303607d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f22daec740b421ab09c01da42631470" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:54:40.330967 12820 consensus_queue.cc:1035] T 6e67e8c44f764aba8de65f51823c3866 P 91e8b37d2bcc4041a11630f9f303607d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bcd53f3752d4d80bcfcbba0c194d9a9" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 39331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:54:40.333372 12828 mvcc.cc:205] Tried to move back new op lower bound from 7334622332231348224 to 7334622332183478272. Current Snapshot: MvccSnapshot[applied={T|T < 7334622332231348224}]
I20260929 10:54:40.337188 12308 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:54:40.337378 12308 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:54:40.338208 12308 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c26351720c0f48109cd9a0bb30c62f84: Sending DeleteTablet for 3 replicas of tablet 6e67e8c44f764aba8de65f51823c3866
I20260929 10:54:40.338559 12727 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e67e8c44f764aba8de65f51823c3866 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:54:40 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:55476
I20260929 10:54:40.338563 12447 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e67e8c44f764aba8de65f51823c3866 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:54:40 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:36640
I20260929 10:54:40.338817 12837 tablet_replica.cc:330] T 6e67e8c44f764aba8de65f51823c3866 P 91e8b37d2bcc4041a11630f9f303607d: stopping tablet replica
I20260929 10:54:40.338845 12587 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e67e8c44f764aba8de65f51823c3866 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:54:40 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:51546
I20260929 10:54:40.338922 12837 raft_consensus.cc:2229] T 6e67e8c44f764aba8de65f51823c3866 P 91e8b37d2bcc4041a11630f9f303607d [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:54:40.339046 12837 raft_consensus.cc:2258] T 6e67e8c44f764aba8de65f51823c3866 P 91e8b37d2bcc4041a11630f9f303607d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:54:40.339231 12838 tablet_replica.cc:330] T 6e67e8c44f764aba8de65f51823c3866 P 8f22daec740b421ab09c01da42631470: stopping tablet replica
I20260929 10:54:40.339243 12308 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38618:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 10:54:40.339339 12838 raft_consensus.cc:2229] T 6e67e8c44f764aba8de65f51823c3866 P 8f22daec740b421ab09c01da42631470 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260929 10:54:40.339375 12308 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:54:40.339710 12838 raft_consensus.cc:2258] T 6e67e8c44f764aba8de65f51823c3866 P 8f22daec740b421ab09c01da42631470 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:54:40.339886 12837 ts_tablet_manager.cc:1905] T 6e67e8c44f764aba8de65f51823c3866 P 91e8b37d2bcc4041a11630f9f303607d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:54:40.339953 12836 tablet_replica.cc:330] T 6e67e8c44f764aba8de65f51823c3866 P 9bcd53f3752d4d80bcfcbba0c194d9a9: stopping tablet replica
I20260929 10:54:40.340032 12836 raft_consensus.cc:2229] T 6e67e8c44f764aba8de65f51823c3866 P 9bcd53f3752d4d80bcfcbba0c194d9a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:54:40.340227 12838 ts_tablet_manager.cc:1905] T 6e67e8c44f764aba8de65f51823c3866 P 8f22daec740b421ab09c01da42631470: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:54:40.340373 12836 raft_consensus.cc:2258] T 6e67e8c44f764aba8de65f51823c3866 P 9bcd53f3752d4d80bcfcbba0c194d9a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:54:40.340791 12836 ts_tablet_manager.cc:1905] T 6e67e8c44f764aba8de65f51823c3866 P 9bcd53f3752d4d80bcfcbba0c194d9a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:54:40.341009 12587 tablet_service.cc:1467] Processing CreateTablet for tablet 61707cf1fa1a418ea716d255998afde3 (DEFAULT_TABLE table=test-table [id=be2e09ee00f74a08bb056227cc8db1b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:54:40.341215 12587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61707cf1fa1a418ea716d255998afde3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:54:40.341671 12837 ts_tablet_manager.cc:1918] T 6e67e8c44f764aba8de65f51823c3866 P 91e8b37d2bcc4041a11630f9f303607d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:54:40.341744 12837 log.cc:1196] T 6e67e8c44f764aba8de65f51823c3866 P 91e8b37d2bcc4041a11630f9f303607d: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-0/wal/wals/6e67e8c44f764aba8de65f51823c3866
I20260929 10:54:40.341820 12838 ts_tablet_manager.cc:1918] T 6e67e8c44f764aba8de65f51823c3866 P 8f22daec740b421ab09c01da42631470: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:54:40.341890 12838 log.cc:1196] T 6e67e8c44f764aba8de65f51823c3866 P 8f22daec740b421ab09c01da42631470: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-1/wal/wals/6e67e8c44f764aba8de65f51823c3866
I20260929 10:54:40.342015 12837 ts_tablet_manager.cc:1939] T 6e67e8c44f764aba8de65f51823c3866 P 91e8b37d2bcc4041a11630f9f303607d: Deleting consensus metadata
I20260929 10:54:40.342185 12838 ts_tablet_manager.cc:1939] T 6e67e8c44f764aba8de65f51823c3866 P 8f22daec740b421ab09c01da42631470: Deleting consensus metadata
I20260929 10:54:40.342403 12295 catalog_manager.cc:4841] TS 91e8b37d2bcc4041a11630f9f303607d (127.10.238.193:40549): tablet 6e67e8c44f764aba8de65f51823c3866 (table test-table [id=e04ebb5123324c7f9a0391fa57ae0e1c]) successfully deleted
I20260929 10:54:40.342584 12297 catalog_manager.cc:4841] TS 8f22daec740b421ab09c01da42631470 (127.10.238.194:40521): tablet 6e67e8c44f764aba8de65f51823c3866 (table test-table [id=e04ebb5123324c7f9a0391fa57ae0e1c]) successfully deleted
I20260929 10:54:40.342657 12814 tablet_bootstrap.cc:492] T 61707cf1fa1a418ea716d255998afde3 P 8f22daec740b421ab09c01da42631470: Bootstrap starting.
I20260929 10:54:40.342701 12447 tablet_service.cc:1467] Processing CreateTablet for tablet 61707cf1fa1a418ea716d255998afde3 (DEFAULT_TABLE table=test-table [id=be2e09ee00f74a08bb056227cc8db1b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:54:40.342787 12447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61707cf1fa1a418ea716d255998afde3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:54:40.343091 12814 tablet_bootstrap.cc:654] T 61707cf1fa1a418ea716d255998afde3 P 8f22daec740b421ab09c01da42631470: Neither blocks nor log segments found. Creating new log.
I20260929 10:54:40.343825 12814 tablet_bootstrap.cc:492] T 61707cf1fa1a418ea716d255998afde3 P 8f22daec740b421ab09c01da42631470: No bootstrap required, opened a new log
I20260929 10:54:40.343876 12814 ts_tablet_manager.cc:1397] T 61707cf1fa1a418ea716d255998afde3 P 8f22daec740b421ab09c01da42631470: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 10:54:40.344028 12814 raft_consensus.cc:348] T 61707cf1fa1a418ea716d255998afde3 P 8f22daec740b421ab09c01da42631470 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e8b37d2bcc4041a11630f9f303607d" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 40549 } } peers { permanent_uuid: "8f22daec740b421ab09c01da42631470" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40521 } } peers { permanent_uuid: "9bcd53f3752d4d80bcfcbba0c194d9a9" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 39331 } }
I20260929 10:54:40.344094 12815 tablet_bootstrap.cc:492] T 61707cf1fa1a418ea716d255998afde3 P 91e8b37d2bcc4041a11630f9f303607d: Bootstrap starting.
I20260929 10:54:40.344099 12814 raft_consensus.cc:374] T 61707cf1fa1a418ea716d255998afde3 P 8f22daec740b421ab09c01da42631470 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:54:40.344121 12814 raft_consensus.cc:729] T 61707cf1fa1a418ea716d255998afde3 P 8f22daec740b421ab09c01da42631470 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f22daec740b421ab09c01da42631470, State: Initialized, Role: FOLLOWER
I20260929 10:54:40.344161 12814 consensus_queue.cc:260] T 61707cf1fa1a418ea716d255998afde3 P 8f22daec740b421ab09c01da42631470 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e8b37d2bcc4041a11630f9f303607d" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 40549 } } peers { permanent_uuid: "8f22daec740b421ab09c01da42631470" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40521 } } peers { permanent_uuid: "9bcd53f3752d4d80bcfcbba0c194d9a9" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 39331 } }
I20260929 10:54:40.344250 12814 ts_tablet_manager.cc:1428] T 61707cf1fa1a418ea716d255998afde3 P 8f22daec740b421ab09c01da42631470: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:40.344486 12815 tablet_bootstrap.cc:654] T 61707cf1fa1a418ea716d255998afde3 P 91e8b37d2bcc4041a11630f9f303607d: Neither blocks nor log segments found. Creating new log.
I20260929 10:54:40.345424 12815 tablet_bootstrap.cc:492] T 61707cf1fa1a418ea716d255998afde3 P 91e8b37d2bcc4041a11630f9f303607d: No bootstrap required, opened a new log
I20260929 10:54:40.345463 12815 ts_tablet_manager.cc:1397] T 61707cf1fa1a418ea716d255998afde3 P 91e8b37d2bcc4041a11630f9f303607d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 10:54:40.345580 12815 raft_consensus.cc:348] T 61707cf1fa1a418ea716d255998afde3 P 91e8b37d2bcc4041a11630f9f303607d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e8b37d2bcc4041a11630f9f303607d" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 40549 } } peers { permanent_uuid: "8f22daec740b421ab09c01da42631470" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40521 } } peers { permanent_uuid: "9bcd53f3752d4d80bcfcbba0c194d9a9" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 39331 } }
I20260929 10:54:40.345636 12815 raft_consensus.cc:374] T 61707cf1fa1a418ea716d255998afde3 P 91e8b37d2bcc4041a11630f9f303607d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:54:40.345649 12815 raft_consensus.cc:729] T 61707cf1fa1a418ea716d255998afde3 P 91e8b37d2bcc4041a11630f9f303607d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91e8b37d2bcc4041a11630f9f303607d, State: Initialized, Role: FOLLOWER
I20260929 10:54:40.345703 12815 consensus_queue.cc:260] T 61707cf1fa1a418ea716d255998afde3 P 91e8b37d2bcc4041a11630f9f303607d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e8b37d2bcc4041a11630f9f303607d" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 40549 } } peers { permanent_uuid: "8f22daec740b421ab09c01da42631470" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40521 } } peers { permanent_uuid: "9bcd53f3752d4d80bcfcbba0c194d9a9" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 39331 } }
I20260929 10:54:40.345794 12815 ts_tablet_manager.cc:1428] T 61707cf1fa1a418ea716d255998afde3 P 91e8b37d2bcc4041a11630f9f303607d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:40.345139 12727 tablet_service.cc:1467] Processing CreateTablet for tablet 61707cf1fa1a418ea716d255998afde3 (DEFAULT_TABLE table=test-table [id=be2e09ee00f74a08bb056227cc8db1b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:54:40.346019 12727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61707cf1fa1a418ea716d255998afde3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:54:40.347414 12813 tablet_bootstrap.cc:492] T 61707cf1fa1a418ea716d255998afde3 P 9bcd53f3752d4d80bcfcbba0c194d9a9: Bootstrap starting.
I20260929 10:54:40.347573 12836 ts_tablet_manager.cc:1918] T 6e67e8c44f764aba8de65f51823c3866 P 9bcd53f3752d4d80bcfcbba0c194d9a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:54:40.347638 12836 log.cc:1196] T 6e67e8c44f764aba8de65f51823c3866 P 9bcd53f3752d4d80bcfcbba0c194d9a9: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-2/wal/wals/6e67e8c44f764aba8de65f51823c3866
I20260929 10:54:40.347827 12813 tablet_bootstrap.cc:654] T 61707cf1fa1a418ea716d255998afde3 P 9bcd53f3752d4d80bcfcbba0c194d9a9: Neither blocks nor log segments found. Creating new log.
I20260929 10:54:40.348052 12836 ts_tablet_manager.cc:1939] T 6e67e8c44f764aba8de65f51823c3866 P 9bcd53f3752d4d80bcfcbba0c194d9a9: Deleting consensus metadata
I20260929 10:54:40.348445 12297 catalog_manager.cc:4841] TS 9bcd53f3752d4d80bcfcbba0c194d9a9 (127.10.238.195:39331): tablet 6e67e8c44f764aba8de65f51823c3866 (table test-table [id=e04ebb5123324c7f9a0391fa57ae0e1c]) successfully deleted
I20260929 10:54:40.348666 12813 tablet_bootstrap.cc:492] T 61707cf1fa1a418ea716d255998afde3 P 9bcd53f3752d4d80bcfcbba0c194d9a9: No bootstrap required, opened a new log
I20260929 10:54:40.348706 12813 ts_tablet_manager.cc:1397] T 61707cf1fa1a418ea716d255998afde3 P 9bcd53f3752d4d80bcfcbba0c194d9a9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 10:54:40.348870 12813 raft_consensus.cc:348] T 61707cf1fa1a418ea716d255998afde3 P 9bcd53f3752d4d80bcfcbba0c194d9a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e8b37d2bcc4041a11630f9f303607d" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 40549 } } peers { permanent_uuid: "8f22daec740b421ab09c01da42631470" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40521 } } peers { permanent_uuid: "9bcd53f3752d4d80bcfcbba0c194d9a9" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 39331 } }
I20260929 10:54:40.348924 12813 raft_consensus.cc:374] T 61707cf1fa1a418ea716d255998afde3 P 9bcd53f3752d4d80bcfcbba0c194d9a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:54:40.348938 12813 raft_consensus.cc:729] T 61707cf1fa1a418ea716d255998afde3 P 9bcd53f3752d4d80bcfcbba0c194d9a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bcd53f3752d4d80bcfcbba0c194d9a9, State: Initialized, Role: FOLLOWER
I20260929 10:54:40.348992 12813 consensus_queue.cc:260] T 61707cf1fa1a418ea716d255998afde3 P 9bcd53f3752d4d80bcfcbba0c194d9a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e8b37d2bcc4041a11630f9f303607d" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 40549 } } peers { permanent_uuid: "8f22daec740b421ab09c01da42631470" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40521 } } peers { permanent_uuid: "9bcd53f3752d4d80bcfcbba0c194d9a9" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 39331 } }
I20260929 10:54:40.349085 12813 ts_tablet_manager.cc:1428] T 61707cf1fa1a418ea716d255998afde3 P 9bcd53f3752d4d80bcfcbba0c194d9a9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260929 10:54:40.393838 12794 tablet.cc:2369] T 61707cf1fa1a418ea716d255998afde3 P 9bcd53f3752d4d80bcfcbba0c194d9a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:54:40.573098 12821 raft_consensus.cc:482] T 61707cf1fa1a418ea716d255998afde3 P 8f22daec740b421ab09c01da42631470 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:54:40.573194 12821 raft_consensus.cc:504] T 61707cf1fa1a418ea716d255998afde3 P 8f22daec740b421ab09c01da42631470 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e8b37d2bcc4041a11630f9f303607d" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 40549 } } peers { permanent_uuid: "8f22daec740b421ab09c01da42631470" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40521 } } peers { permanent_uuid: "9bcd53f3752d4d80bcfcbba0c194d9a9" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 39331 } }
I20260929 10:54:40.573678 12821 leader_election.cc:288] T 61707cf1fa1a418ea716d255998afde3 P 8f22daec740b421ab09c01da42631470 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91e8b37d2bcc4041a11630f9f303607d (127.10.238.193:40549), 9bcd53f3752d4d80bcfcbba0c194d9a9 (127.10.238.195:39331)
I20260929 10:54:40.576630 12810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:40.573942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:47415 (local address 127.10.238.195:39331)
0929 10:54:40.573992 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:40.573995 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:40.574132 (+ 137us) server_negotiation.cc:407] Connection header received
0929 10:54:40.574201 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:40.574202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:40.574211 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:40.574293 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:54:40.574443 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.574907 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.575906 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.576257 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.576357 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:40.576365 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:40.576523 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 10:54:40.576563 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 10:54:40.576637 12809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:40.573970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:53581 (local address 127.10.238.193:40549)
0929 10:54:40.574054 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:40.574056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:40.574067 (+ 11us) server_negotiation.cc:407] Connection header received
0929 10:54:40.574164 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:40.574165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:40.574185 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:40.574269 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:54:40.574434 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.574909 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.575899 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.576268 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.576363 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:40.576371 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:40.576530 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 10:54:40.576570 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 10:54:40.576647 12842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:40.573865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.195:39331 (local address 127.10.238.194:47415)
0929 10:54:40.574054 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:40.574063 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:40.574142 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:40.574300 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:40.574300 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:40.574303 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:54:40.574373 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:40.574375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.574992 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.574993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:40.575811 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:40.575813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.576357 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.576358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:40.576437 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:40.576440 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 10:54:40.576502 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 10:54:40.576547 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":46,"thread_start_us":65,"threads_started":1}
I20260929 10:54:40.576647 12843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:40.573865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:40549 (local address 127.10.238.194:53581)
0929 10:54:40.574010 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:40.574023 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:40.574082 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:40.574277 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:40.574278 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:40.574280 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:54:40.574360 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:40.574362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.574994 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.574995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:40.575801 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:40.575803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.576362 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.576363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:40.576446 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:40.576449 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 10:54:40.576508 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 10:54:40.576552 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":64,"threads_started":1}
I20260929 10:54:40.576960 12747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61707cf1fa1a418ea716d255998afde3" candidate_uuid: "8f22daec740b421ab09c01da42631470" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bcd53f3752d4d80bcfcbba0c194d9a9" is_pre_election: true
I20260929 10:54:40.576988 12467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61707cf1fa1a418ea716d255998afde3" candidate_uuid: "8f22daec740b421ab09c01da42631470" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91e8b37d2bcc4041a11630f9f303607d" is_pre_election: true
I20260929 10:54:40.577062 12747 raft_consensus.cc:2454] T 61707cf1fa1a418ea716d255998afde3 P 9bcd53f3752d4d80bcfcbba0c194d9a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f22daec740b421ab09c01da42631470 in term 0.
I20260929 10:54:40.577104 12467 raft_consensus.cc:2454] T 61707cf1fa1a418ea716d255998afde3 P 91e8b37d2bcc4041a11630f9f303607d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f22daec740b421ab09c01da42631470 in term 0.
I20260929 10:54:40.577240 12541 leader_election.cc:302] T 61707cf1fa1a418ea716d255998afde3 P 8f22daec740b421ab09c01da42631470 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f22daec740b421ab09c01da42631470, 9bcd53f3752d4d80bcfcbba0c194d9a9; no voters:
I20260929 10:54:40.577391 12821 raft_consensus.cc:2781] T 61707cf1fa1a418ea716d255998afde3 P 8f22daec740b421ab09c01da42631470 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:54:40.577427 12821 raft_consensus.cc:482] T 61707cf1fa1a418ea716d255998afde3 P 8f22daec740b421ab09c01da42631470 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:54:40.577451 12821 raft_consensus.cc:3037] T 61707cf1fa1a418ea716d255998afde3 P 8f22daec740b421ab09c01da42631470 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:54:40.578123 12821 raft_consensus.cc:504] T 61707cf1fa1a418ea716d255998afde3 P 8f22daec740b421ab09c01da42631470 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e8b37d2bcc4041a11630f9f303607d" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 40549 } } peers { permanent_uuid: "8f22daec740b421ab09c01da42631470" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40521 } } peers { permanent_uuid: "9bcd53f3752d4d80bcfcbba0c194d9a9" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 39331 } }
I20260929 10:54:40.578269 12821 leader_election.cc:288] T 61707cf1fa1a418ea716d255998afde3 P 8f22daec740b421ab09c01da42631470 [CANDIDATE]: Term 1 election: Requested vote from peers 91e8b37d2bcc4041a11630f9f303607d (127.10.238.193:40549), 9bcd53f3752d4d80bcfcbba0c194d9a9 (127.10.238.195:39331)
I20260929 10:54:40.578400 12467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61707cf1fa1a418ea716d255998afde3" candidate_uuid: "8f22daec740b421ab09c01da42631470" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91e8b37d2bcc4041a11630f9f303607d"
I20260929 10:54:40.578421 12747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61707cf1fa1a418ea716d255998afde3" candidate_uuid: "8f22daec740b421ab09c01da42631470" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bcd53f3752d4d80bcfcbba0c194d9a9"
I20260929 10:54:40.578491 12747 raft_consensus.cc:3037] T 61707cf1fa1a418ea716d255998afde3 P 9bcd53f3752d4d80bcfcbba0c194d9a9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:54:40.578491 12467 raft_consensus.cc:3037] T 61707cf1fa1a418ea716d255998afde3 P 91e8b37d2bcc4041a11630f9f303607d [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:54:40.579099 12747 raft_consensus.cc:2454] T 61707cf1fa1a418ea716d255998afde3 P 9bcd53f3752d4d80bcfcbba0c194d9a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f22daec740b421ab09c01da42631470 in term 1.
I20260929 10:54:40.579188 12467 raft_consensus.cc:2454] T 61707cf1fa1a418ea716d255998afde3 P 91e8b37d2bcc4041a11630f9f303607d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f22daec740b421ab09c01da42631470 in term 1.
I20260929 10:54:40.579257 12541 leader_election.cc:302] T 61707cf1fa1a418ea716d255998afde3 P 8f22daec740b421ab09c01da42631470 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f22daec740b421ab09c01da42631470, 9bcd53f3752d4d80bcfcbba0c194d9a9; no voters:
I20260929 10:54:40.579347 12821 raft_consensus.cc:2781] T 61707cf1fa1a418ea716d255998afde3 P 8f22daec740b421ab09c01da42631470 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:54:40.579449 12821 raft_consensus.cc:686] T 61707cf1fa1a418ea716d255998afde3 P 8f22daec740b421ab09c01da42631470 [term 1 LEADER]: Becoming Leader. State: Replica: 8f22daec740b421ab09c01da42631470, State: Running, Role: LEADER
I20260929 10:54:40.579504 12821 consensus_queue.cc:237] T 61707cf1fa1a418ea716d255998afde3 P 8f22daec740b421ab09c01da42631470 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e8b37d2bcc4041a11630f9f303607d" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 40549 } } peers { permanent_uuid: "8f22daec740b421ab09c01da42631470" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40521 } } peers { permanent_uuid: "9bcd53f3752d4d80bcfcbba0c194d9a9" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 39331 } }
I20260929 10:54:40.580191 12308 catalog_manager.cc:5495] T 61707cf1fa1a418ea716d255998afde3 P 8f22daec740b421ab09c01da42631470 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f22daec740b421ab09c01da42631470 (127.10.238.194). New cstate: current_term: 1 leader_uuid: "8f22daec740b421ab09c01da42631470" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e8b37d2bcc4041a11630f9f303607d" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 40549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f22daec740b421ab09c01da42631470" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bcd53f3752d4d80bcfcbba0c194d9a9" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 39331 } health_report { overall_health: UNKNOWN } } }
W20260929 10:54:40.585070 12514 tablet.cc:2369] T 61707cf1fa1a418ea716d255998afde3 P 91e8b37d2bcc4041a11630f9f303607d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 10:54:40 dist-test-slave-8nrt krb5kdc[12265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679279, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:54:40.644786 12359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:40.638001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38632 (local address 127.10.238.254:41855)
0929 10:54:40.638083 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:40.638085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:40.638164 (+ 79us) server_negotiation.cc:407] Connection header received
0929 10:54:40.638205 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:40.638206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:40.638218 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:40.638280 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:40.638753 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.639228 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.639894 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.640075 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.640129 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:40.642190 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:40.642191 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:40.642191 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:40.642197 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:40.643574 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:40.643935 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:40.643937 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:40.643937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:40.644010 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:40.644230 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:40.644231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:40.644231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:40.644338 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:40.644431 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:40.644580 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 10:54:40.644662 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 10:54:40.644865 12851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:40.637930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:41855 (local address 127.0.0.1:38632)
0929 10:54:40.638087 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:40.638106 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:40.638163 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:40.638305 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:40.638306 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:40.638598 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:54:40.638645 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:40.638647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.639300 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.639301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:40.639823 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:40.639824 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.640142 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.640142 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:40.640183 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:40.640690 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:54:40.640694 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:54:40.642106 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:54:40.643667 (+ 1561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:40.643674 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:40.643677 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:40.643861 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:40.644074 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:40.644075 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:40.644076 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:40.644157 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:40.644423 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:54:40.644430 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:54:40.644482 (+ 52us) client_negotiation.cc:769] Sending connection context
0929 10:54:40.644556 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 10:54:40.644717 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":66,"thread_start_us":86,"threads_started":1}
I20260929 10:54:40.647177 12308 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:38632:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 10:54:40.647396 12308 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:54:40.648787 12587 tablet_service.cc:1467] Processing CreateTablet for tablet f73f47df02dc4f8f9e19eae07297c8c6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6e8faece9c354522a6ca9d01207527cb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 10:54:40.648835 12727 tablet_service.cc:1467] Processing CreateTablet for tablet f73f47df02dc4f8f9e19eae07297c8c6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6e8faece9c354522a6ca9d01207527cb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 10:54:40.648973 12587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f73f47df02dc4f8f9e19eae07297c8c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:54:40.648998 12727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f73f47df02dc4f8f9e19eae07297c8c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:54:40.648757 12447 tablet_service.cc:1467] Processing CreateTablet for tablet f73f47df02dc4f8f9e19eae07297c8c6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6e8faece9c354522a6ca9d01207527cb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 10:54:40.651005 12447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f73f47df02dc4f8f9e19eae07297c8c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:54:40.651193 12813 tablet_bootstrap.cc:492] T f73f47df02dc4f8f9e19eae07297c8c6 P 9bcd53f3752d4d80bcfcbba0c194d9a9: Bootstrap starting.
I20260929 10:54:40.651592 12813 tablet_bootstrap.cc:654] T f73f47df02dc4f8f9e19eae07297c8c6 P 9bcd53f3752d4d80bcfcbba0c194d9a9: Neither blocks nor log segments found. Creating new log.
I20260929 10:54:40.652017 12815 tablet_bootstrap.cc:492] T f73f47df02dc4f8f9e19eae07297c8c6 P 91e8b37d2bcc4041a11630f9f303607d: Bootstrap starting.
I20260929 10:54:40.652367 12815 tablet_bootstrap.cc:654] T f73f47df02dc4f8f9e19eae07297c8c6 P 91e8b37d2bcc4041a11630f9f303607d: Neither blocks nor log segments found. Creating new log.
I20260929 10:54:40.652913 12815 tablet_bootstrap.cc:492] T f73f47df02dc4f8f9e19eae07297c8c6 P 91e8b37d2bcc4041a11630f9f303607d: No bootstrap required, opened a new log
I20260929 10:54:40.652976 12815 ts_tablet_manager.cc:1397] T f73f47df02dc4f8f9e19eae07297c8c6 P 91e8b37d2bcc4041a11630f9f303607d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260929 10:54:40.653100 12815 raft_consensus.cc:348] T f73f47df02dc4f8f9e19eae07297c8c6 P 91e8b37d2bcc4041a11630f9f303607d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e8b37d2bcc4041a11630f9f303607d" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 40549 } } peers { permanent_uuid: "8f22daec740b421ab09c01da42631470" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40521 } } peers { permanent_uuid: "9bcd53f3752d4d80bcfcbba0c194d9a9" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 39331 } }
I20260929 10:54:40.653172 12815 raft_consensus.cc:374] T f73f47df02dc4f8f9e19eae07297c8c6 P 91e8b37d2bcc4041a11630f9f303607d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:54:40.653205 12815 raft_consensus.cc:729] T f73f47df02dc4f8f9e19eae07297c8c6 P 91e8b37d2bcc4041a11630f9f303607d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91e8b37d2bcc4041a11630f9f303607d, State: Initialized, Role: FOLLOWER
I20260929 10:54:40.653283 12815 consensus_queue.cc:260] T f73f47df02dc4f8f9e19eae07297c8c6 P 91e8b37d2bcc4041a11630f9f303607d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e8b37d2bcc4041a11630f9f303607d" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 40549 } } peers { permanent_uuid: "8f22daec740b421ab09c01da42631470" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40521 } } peers { permanent_uuid: "9bcd53f3752d4d80bcfcbba0c194d9a9" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 39331 } }
I20260929 10:54:40.653391 12815 ts_tablet_manager.cc:1428] T f73f47df02dc4f8f9e19eae07297c8c6 P 91e8b37d2bcc4041a11630f9f303607d: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260929 10:54:40.653414 12831 tablet_replica.cc:439] T f73f47df02dc4f8f9e19eae07297c8c6 P 91e8b37d2bcc4041a11630f9f303607d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e8b37d2bcc4041a11630f9f303607d" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 40549 } } peers { permanent_uuid: "8f22daec740b421ab09c01da42631470" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40521 } } peers { permanent_uuid: "9bcd53f3752d4d80bcfcbba0c194d9a9" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 39331 } } }
I20260929 10:54:40.653519 12831 tablet_replica.cc:442] T f73f47df02dc4f8f9e19eae07297c8c6 P 91e8b37d2bcc4041a11630f9f303607d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:54:40.653541 12814 tablet_bootstrap.cc:492] T f73f47df02dc4f8f9e19eae07297c8c6 P 8f22daec740b421ab09c01da42631470: Bootstrap starting.
I20260929 10:54:40.653864 12813 tablet_bootstrap.cc:492] T f73f47df02dc4f8f9e19eae07297c8c6 P 9bcd53f3752d4d80bcfcbba0c194d9a9: No bootstrap required, opened a new log
I20260929 10:54:40.653932 12813 ts_tablet_manager.cc:1397] T f73f47df02dc4f8f9e19eae07297c8c6 P 9bcd53f3752d4d80bcfcbba0c194d9a9: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 10:54:40.654003 12814 tablet_bootstrap.cc:654] T f73f47df02dc4f8f9e19eae07297c8c6 P 8f22daec740b421ab09c01da42631470: Neither blocks nor log segments found. Creating new log.
I20260929 10:54:40.654083 12813 raft_consensus.cc:348] T f73f47df02dc4f8f9e19eae07297c8c6 P 9bcd53f3752d4d80bcfcbba0c194d9a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e8b37d2bcc4041a11630f9f303607d" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 40549 } } peers { permanent_uuid: "8f22daec740b421ab09c01da42631470" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40521 } } peers { permanent_uuid: "9bcd53f3752d4d80bcfcbba0c194d9a9" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 39331 } }
I20260929 10:54:40.654134 12813 raft_consensus.cc:374] T f73f47df02dc4f8f9e19eae07297c8c6 P 9bcd53f3752d4d80bcfcbba0c194d9a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:54:40.654145 12813 raft_consensus.cc:729] T f73f47df02dc4f8f9e19eae07297c8c6 P 9bcd53f3752d4d80bcfcbba0c194d9a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bcd53f3752d4d80bcfcbba0c194d9a9, State: Initialized, Role: FOLLOWER
I20260929 10:54:40.654177 12813 consensus_queue.cc:260] T f73f47df02dc4f8f9e19eae07297c8c6 P 9bcd53f3752d4d80bcfcbba0c194d9a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e8b37d2bcc4041a11630f9f303607d" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 40549 } } peers { permanent_uuid: "8f22daec740b421ab09c01da42631470" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40521 } } peers { permanent_uuid: "9bcd53f3752d4d80bcfcbba0c194d9a9" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 39331 } }
I20260929 10:54:40.654276 12813 ts_tablet_manager.cc:1428] T f73f47df02dc4f8f9e19eae07297c8c6 P 9bcd53f3752d4d80bcfcbba0c194d9a9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:40.654326 12819 tablet_replica.cc:439] T f73f47df02dc4f8f9e19eae07297c8c6 P 9bcd53f3752d4d80bcfcbba0c194d9a9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e8b37d2bcc4041a11630f9f303607d" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 40549 } } peers { permanent_uuid: "8f22daec740b421ab09c01da42631470" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40521 } } peers { permanent_uuid: "9bcd53f3752d4d80bcfcbba0c194d9a9" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 39331 } } }
I20260929 10:54:40.654412 12819 tablet_replica.cc:442] T f73f47df02dc4f8f9e19eae07297c8c6 P 9bcd53f3752d4d80bcfcbba0c194d9a9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:54:40.654794 12814 tablet_bootstrap.cc:492] T f73f47df02dc4f8f9e19eae07297c8c6 P 8f22daec740b421ab09c01da42631470: No bootstrap required, opened a new log
I20260929 10:54:40.654860 12814 ts_tablet_manager.cc:1397] T f73f47df02dc4f8f9e19eae07297c8c6 P 8f22daec740b421ab09c01da42631470: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 10:54:40.655018 12814 raft_consensus.cc:348] T f73f47df02dc4f8f9e19eae07297c8c6 P 8f22daec740b421ab09c01da42631470 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e8b37d2bcc4041a11630f9f303607d" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 40549 } } peers { permanent_uuid: "8f22daec740b421ab09c01da42631470" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40521 } } peers { permanent_uuid: "9bcd53f3752d4d80bcfcbba0c194d9a9" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 39331 } }
I20260929 10:54:40.655092 12814 raft_consensus.cc:374] T f73f47df02dc4f8f9e19eae07297c8c6 P 8f22daec740b421ab09c01da42631470 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:54:40.655124 12814 raft_consensus.cc:729] T f73f47df02dc4f8f9e19eae07297c8c6 P 8f22daec740b421ab09c01da42631470 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f22daec740b421ab09c01da42631470, State: Initialized, Role: FOLLOWER
I20260929 10:54:40.655172 12814 consensus_queue.cc:260] T f73f47df02dc4f8f9e19eae07297c8c6 P 8f22daec740b421ab09c01da42631470 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e8b37d2bcc4041a11630f9f303607d" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 40549 } } peers { permanent_uuid: "8f22daec740b421ab09c01da42631470" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40521 } } peers { permanent_uuid: "9bcd53f3752d4d80bcfcbba0c194d9a9" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 39331 } }
I20260929 10:54:40.655267 12814 ts_tablet_manager.cc:1428] T f73f47df02dc4f8f9e19eae07297c8c6 P 8f22daec740b421ab09c01da42631470: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 10:54:40.655301 12821 tablet_replica.cc:439] T f73f47df02dc4f8f9e19eae07297c8c6 P 8f22daec740b421ab09c01da42631470: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e8b37d2bcc4041a11630f9f303607d" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 40549 } } peers { permanent_uuid: "8f22daec740b421ab09c01da42631470" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40521 } } peers { permanent_uuid: "9bcd53f3752d4d80bcfcbba0c194d9a9" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 39331 } } }
I20260929 10:54:40.655411 12821 tablet_replica.cc:442] T f73f47df02dc4f8f9e19eae07297c8c6 P 8f22daec740b421ab09c01da42631470: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:54:40.880040 12821 raft_consensus.cc:482] T f73f47df02dc4f8f9e19eae07297c8c6 P 8f22daec740b421ab09c01da42631470 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:54:40.880133 12821 raft_consensus.cc:504] T f73f47df02dc4f8f9e19eae07297c8c6 P 8f22daec740b421ab09c01da42631470 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e8b37d2bcc4041a11630f9f303607d" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 40549 } } peers { permanent_uuid: "8f22daec740b421ab09c01da42631470" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40521 } } peers { permanent_uuid: "9bcd53f3752d4d80bcfcbba0c194d9a9" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 39331 } }
I20260929 10:54:40.880286 12821 leader_election.cc:288] T f73f47df02dc4f8f9e19eae07297c8c6 P 8f22daec740b421ab09c01da42631470 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91e8b37d2bcc4041a11630f9f303607d (127.10.238.193:40549), 9bcd53f3752d4d80bcfcbba0c194d9a9 (127.10.238.195:39331)
I20260929 10:54:40.880479 12747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73f47df02dc4f8f9e19eae07297c8c6" candidate_uuid: "8f22daec740b421ab09c01da42631470" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bcd53f3752d4d80bcfcbba0c194d9a9" is_pre_election: true
I20260929 10:54:40.880590 12747 raft_consensus.cc:2454] T f73f47df02dc4f8f9e19eae07297c8c6 P 9bcd53f3752d4d80bcfcbba0c194d9a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f22daec740b421ab09c01da42631470 in term 0.
I20260929 10:54:40.880530 12467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73f47df02dc4f8f9e19eae07297c8c6" candidate_uuid: "8f22daec740b421ab09c01da42631470" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91e8b37d2bcc4041a11630f9f303607d" is_pre_election: true
I20260929 10:54:40.880640 12467 raft_consensus.cc:2454] T f73f47df02dc4f8f9e19eae07297c8c6 P 91e8b37d2bcc4041a11630f9f303607d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f22daec740b421ab09c01da42631470 in term 0.
I20260929 10:54:40.880786 12541 leader_election.cc:302] T f73f47df02dc4f8f9e19eae07297c8c6 P 8f22daec740b421ab09c01da42631470 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f22daec740b421ab09c01da42631470, 9bcd53f3752d4d80bcfcbba0c194d9a9; no voters:
I20260929 10:54:40.880909 12821 raft_consensus.cc:2781] T f73f47df02dc4f8f9e19eae07297c8c6 P 8f22daec740b421ab09c01da42631470 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:54:40.880976 12821 raft_consensus.cc:482] T f73f47df02dc4f8f9e19eae07297c8c6 P 8f22daec740b421ab09c01da42631470 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:54:40.881011 12821 raft_consensus.cc:3037] T f73f47df02dc4f8f9e19eae07297c8c6 P 8f22daec740b421ab09c01da42631470 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:54:40.881701 12821 raft_consensus.cc:504] T f73f47df02dc4f8f9e19eae07297c8c6 P 8f22daec740b421ab09c01da42631470 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e8b37d2bcc4041a11630f9f303607d" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 40549 } } peers { permanent_uuid: "8f22daec740b421ab09c01da42631470" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40521 } } peers { permanent_uuid: "9bcd53f3752d4d80bcfcbba0c194d9a9" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 39331 } }
I20260929 10:54:40.881847 12821 leader_election.cc:288] T f73f47df02dc4f8f9e19eae07297c8c6 P 8f22daec740b421ab09c01da42631470 [CANDIDATE]: Term 1 election: Requested vote from peers 91e8b37d2bcc4041a11630f9f303607d (127.10.238.193:40549), 9bcd53f3752d4d80bcfcbba0c194d9a9 (127.10.238.195:39331)
I20260929 10:54:40.881992 12747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73f47df02dc4f8f9e19eae07297c8c6" candidate_uuid: "8f22daec740b421ab09c01da42631470" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bcd53f3752d4d80bcfcbba0c194d9a9"
I20260929 10:54:40.882004 12467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73f47df02dc4f8f9e19eae07297c8c6" candidate_uuid: "8f22daec740b421ab09c01da42631470" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91e8b37d2bcc4041a11630f9f303607d"
I20260929 10:54:40.882087 12467 raft_consensus.cc:3037] T f73f47df02dc4f8f9e19eae07297c8c6 P 91e8b37d2bcc4041a11630f9f303607d [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:54:40.882089 12747 raft_consensus.cc:3037] T f73f47df02dc4f8f9e19eae07297c8c6 P 9bcd53f3752d4d80bcfcbba0c194d9a9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:54:40.882648 12747 raft_consensus.cc:2454] T f73f47df02dc4f8f9e19eae07297c8c6 P 9bcd53f3752d4d80bcfcbba0c194d9a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f22daec740b421ab09c01da42631470 in term 1.
I20260929 10:54:40.882792 12467 raft_consensus.cc:2454] T f73f47df02dc4f8f9e19eae07297c8c6 P 91e8b37d2bcc4041a11630f9f303607d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f22daec740b421ab09c01da42631470 in term 1.
I20260929 10:54:40.882822 12541 leader_election.cc:302] T f73f47df02dc4f8f9e19eae07297c8c6 P 8f22daec740b421ab09c01da42631470 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f22daec740b421ab09c01da42631470, 9bcd53f3752d4d80bcfcbba0c194d9a9; no voters:
I20260929 10:54:40.882893 12821 raft_consensus.cc:2781] T f73f47df02dc4f8f9e19eae07297c8c6 P 8f22daec740b421ab09c01da42631470 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:54:40.882946 12821 raft_consensus.cc:686] T f73f47df02dc4f8f9e19eae07297c8c6 P 8f22daec740b421ab09c01da42631470 [term 1 LEADER]: Becoming Leader. State: Replica: 8f22daec740b421ab09c01da42631470, State: Running, Role: LEADER
I20260929 10:54:40.883003 12821 consensus_queue.cc:237] T f73f47df02dc4f8f9e19eae07297c8c6 P 8f22daec740b421ab09c01da42631470 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e8b37d2bcc4041a11630f9f303607d" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 40549 } } peers { permanent_uuid: "8f22daec740b421ab09c01da42631470" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40521 } } peers { permanent_uuid: "9bcd53f3752d4d80bcfcbba0c194d9a9" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 39331 } }
I20260929 10:54:40.883315 12844 tablet_replica.cc:439] T f73f47df02dc4f8f9e19eae07297c8c6 P 8f22daec740b421ab09c01da42631470: TxnStatusTablet state changed. Reason: New leader 8f22daec740b421ab09c01da42631470. Latest consensus state: current_term: 1 leader_uuid: "8f22daec740b421ab09c01da42631470" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e8b37d2bcc4041a11630f9f303607d" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 40549 } } peers { permanent_uuid: "8f22daec740b421ab09c01da42631470" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40521 } } peers { permanent_uuid: "9bcd53f3752d4d80bcfcbba0c194d9a9" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 39331 } } }
I20260929 10:54:40.883416 12844 tablet_replica.cc:442] T f73f47df02dc4f8f9e19eae07297c8c6 P 8f22daec740b421ab09c01da42631470: This TxnStatusTablet replica's current role is: LEADER
I20260929 10:54:40.883569 12856 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 10:54:40.883587 12308 catalog_manager.cc:5495] T f73f47df02dc4f8f9e19eae07297c8c6 P 8f22daec740b421ab09c01da42631470 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f22daec740b421ab09c01da42631470 (127.10.238.194). New cstate: current_term: 1 leader_uuid: "8f22daec740b421ab09c01da42631470" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e8b37d2bcc4041a11630f9f303607d" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 40549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f22daec740b421ab09c01da42631470" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bcd53f3752d4d80bcfcbba0c194d9a9" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 39331 } health_report { overall_health: UNKNOWN } } }
I20260929 10:54:40.883715 12856 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 10:54:40.938025 12851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:40.935091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.194:40521 (local address 127.0.0.1:51560)
0929 10:54:40.935169 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:40.935181 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:40.935224 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:40.935426 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:40.935427 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:40.935429 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 10:54:40.935483 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:40.935485 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.936107 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.936108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:40.936990 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:40.936993 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.937453 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.937454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:40.937530 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:40.937544 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 10:54:40.937839 (+ 295us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 10:54:40.937849 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 10:54:40.937906 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 10:54:40.937958 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260929 10:54:40.938021 12859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:40.935159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51560 (local address 127.10.238.194:40521)
0929 10:54:40.935256 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:40.935258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:40.935280 (+ 22us) server_negotiation.cc:407] Connection header received
0929 10:54:40.935318 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:40.935319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:40.935334 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:40.935395 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:54:40.935558 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.936026 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.937082 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.937376 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.937453 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:40.937662 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:54:40.937774 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:54:40.937854 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:40.937914 (+ 60us) server_negotiation.cc:299] Negotiation successful
0929 10:54:40.937935 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":55,"threads_started":1}
I20260929 10:54:40.939621 12747 raft_consensus.cc:1261] T f73f47df02dc4f8f9e19eae07297c8c6 P 9bcd53f3752d4d80bcfcbba0c194d9a9 [term 1 FOLLOWER]: Refusing update from remote peer 8f22daec740b421ab09c01da42631470: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:54:40.939736 12467 raft_consensus.cc:1261] T f73f47df02dc4f8f9e19eae07297c8c6 P 91e8b37d2bcc4041a11630f9f303607d [term 1 FOLLOWER]: Refusing update from remote peer 8f22daec740b421ab09c01da42631470: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:54:40.939910 12821 consensus_queue.cc:1035] T f73f47df02dc4f8f9e19eae07297c8c6 P 8f22daec740b421ab09c01da42631470 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bcd53f3752d4d80bcfcbba0c194d9a9" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 39331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:54:40.940022 12844 consensus_queue.cc:1035] T f73f47df02dc4f8f9e19eae07297c8c6 P 8f22daec740b421ab09c01da42631470 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91e8b37d2bcc4041a11630f9f303607d" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 40549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:54:40.940742 12831 tablet_replica.cc:439] T f73f47df02dc4f8f9e19eae07297c8c6 P 91e8b37d2bcc4041a11630f9f303607d: TxnStatusTablet state changed. Reason: New leader 8f22daec740b421ab09c01da42631470. Latest consensus state: current_term: 1 leader_uuid: "8f22daec740b421ab09c01da42631470" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e8b37d2bcc4041a11630f9f303607d" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 40549 } } peers { permanent_uuid: "8f22daec740b421ab09c01da42631470" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40521 } } peers { permanent_uuid: "9bcd53f3752d4d80bcfcbba0c194d9a9" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 39331 } } }
I20260929 10:54:40.940837 12831 tablet_replica.cc:442] T f73f47df02dc4f8f9e19eae07297c8c6 P 91e8b37d2bcc4041a11630f9f303607d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:54:40.940996 12819 tablet_replica.cc:439] T f73f47df02dc4f8f9e19eae07297c8c6 P 9bcd53f3752d4d80bcfcbba0c194d9a9: TxnStatusTablet state changed. Reason: New leader 8f22daec740b421ab09c01da42631470. Latest consensus state: current_term: 1 leader_uuid: "8f22daec740b421ab09c01da42631470" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e8b37d2bcc4041a11630f9f303607d" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 40549 } } peers { permanent_uuid: "8f22daec740b421ab09c01da42631470" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40521 } } peers { permanent_uuid: "9bcd53f3752d4d80bcfcbba0c194d9a9" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 39331 } } }
I20260929 10:54:40.941085 12819 tablet_replica.cc:442] T f73f47df02dc4f8f9e19eae07297c8c6 P 9bcd53f3752d4d80bcfcbba0c194d9a9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:54:40.941558 12819 tablet_replica.cc:439] T f73f47df02dc4f8f9e19eae07297c8c6 P 9bcd53f3752d4d80bcfcbba0c194d9a9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8f22daec740b421ab09c01da42631470" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e8b37d2bcc4041a11630f9f303607d" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 40549 } } peers { permanent_uuid: "8f22daec740b421ab09c01da42631470" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40521 } } peers { permanent_uuid: "9bcd53f3752d4d80bcfcbba0c194d9a9" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 39331 } } }
I20260929 10:54:40.941651 12819 tablet_replica.cc:442] T f73f47df02dc4f8f9e19eae07297c8c6 P 9bcd53f3752d4d80bcfcbba0c194d9a9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:54:40.941694 12821 tablet_replica.cc:439] T f73f47df02dc4f8f9e19eae07297c8c6 P 8f22daec740b421ab09c01da42631470: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8f22daec740b421ab09c01da42631470" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e8b37d2bcc4041a11630f9f303607d" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 40549 } } peers { permanent_uuid: "8f22daec740b421ab09c01da42631470" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40521 } } peers { permanent_uuid: "9bcd53f3752d4d80bcfcbba0c194d9a9" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 39331 } } }
I20260929 10:54:40.941776 12821 tablet_replica.cc:442] T f73f47df02dc4f8f9e19eae07297c8c6 P 8f22daec740b421ab09c01da42631470: This TxnStatusTablet replica's current role is: LEADER
I20260929 10:54:40.942056 12821 tablet_replica.cc:439] T f73f47df02dc4f8f9e19eae07297c8c6 P 8f22daec740b421ab09c01da42631470: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8f22daec740b421ab09c01da42631470" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e8b37d2bcc4041a11630f9f303607d" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 40549 } } peers { permanent_uuid: "8f22daec740b421ab09c01da42631470" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40521 } } peers { permanent_uuid: "9bcd53f3752d4d80bcfcbba0c194d9a9" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 39331 } } }
I20260929 10:54:40.942142 12821 tablet_replica.cc:442] T f73f47df02dc4f8f9e19eae07297c8c6 P 8f22daec740b421ab09c01da42631470: This TxnStatusTablet replica's current role is: LEADER
I20260929 10:54:40.942282 12860 mvcc.cc:205] Tried to move back new op lower bound from 7334622334726553600 to 7334622334497230848. Current Snapshot: MvccSnapshot[applied={T|T < 7334622334726553600}]
I20260929 10:54:40.942471 12831 tablet_replica.cc:439] T f73f47df02dc4f8f9e19eae07297c8c6 P 91e8b37d2bcc4041a11630f9f303607d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8f22daec740b421ab09c01da42631470" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e8b37d2bcc4041a11630f9f303607d" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 40549 } } peers { permanent_uuid: "8f22daec740b421ab09c01da42631470" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40521 } } peers { permanent_uuid: "9bcd53f3752d4d80bcfcbba0c194d9a9" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 39331 } } }
I20260929 10:54:40.942562 12831 tablet_replica.cc:442] T f73f47df02dc4f8f9e19eae07297c8c6 P 91e8b37d2bcc4041a11630f9f303607d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:54:40.942624 12863 mvcc.cc:205] Tried to move back new op lower bound from 7334622334726553600 to 7334622334497230848. Current Snapshot: MvccSnapshot[applied={T|T < 7334622334726553600}]
I20260929 10:54:40.946791 12859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:40.944303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51568 (local address 127.10.238.194:40521)
0929 10:54:40.944342 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:40.944345 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:40.944390 (+ 45us) server_negotiation.cc:407] Connection header received
0929 10:54:40.944460 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:40.944460 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:40.944470 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:40.944524 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:54:40.944702 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.945277 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.946099 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.946322 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.946400 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:40.946539 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:54:40.946573 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:54:40.946624 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:40.946691 (+ 67us) server_negotiation.cc:299] Negotiation successful
0929 10:54:40.946717 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 10:54:40.951550 12859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:40.949237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:55417 (local address 127.10.238.194:40521)
0929 10:54:40.949288 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:40.949290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:40.949422 (+ 132us) server_negotiation.cc:407] Connection header received
0929 10:54:40.949494 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:40.949495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:40.949503 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:40.949546 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:54:40.949699 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.950139 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.950866 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.951046 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.951117 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:40.951303 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:54:40.951339 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:54:40.951394 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:40.951453 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 10:54:40.951471 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 10:54:40.951902 12872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:40.949175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.194:40521 (local address 127.10.238.194:55417)
0929 10:54:40.949354 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:40.949363 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:40.949437 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:40.949558 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:40.949559 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:40.949561 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 10:54:40.949623 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:40.949624 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.950224 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.950225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:40.950776 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:40.950777 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.951135 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.951136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:40.951193 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:40.951216 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 10:54:40.951409 (+ 193us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 10:54:40.951411 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 10:54:40.951452 (+ 41us) client_negotiation.cc:240] Negotiation successful
0929 10:54:40.951491 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":94,"threads_started":1}
I20260929 10:54:40.954214 12747 raft_consensus.cc:1261] T 61707cf1fa1a418ea716d255998afde3 P 9bcd53f3752d4d80bcfcbba0c194d9a9 [term 1 FOLLOWER]: Refusing update from remote peer 8f22daec740b421ab09c01da42631470: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:54:40.954233 12467 raft_consensus.cc:1261] T 61707cf1fa1a418ea716d255998afde3 P 91e8b37d2bcc4041a11630f9f303607d [term 1 FOLLOWER]: Refusing update from remote peer 8f22daec740b421ab09c01da42631470: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:54:40.954509 12873 consensus_queue.cc:1035] T 61707cf1fa1a418ea716d255998afde3 P 8f22daec740b421ab09c01da42631470 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91e8b37d2bcc4041a11630f9f303607d" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 40549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:54:40.954610 12844 consensus_queue.cc:1035] T 61707cf1fa1a418ea716d255998afde3 P 8f22daec740b421ab09c01da42631470 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bcd53f3752d4d80bcfcbba0c194d9a9" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 39331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:54:40.968166 12308 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:54:40.968271 12308 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:54:40.968889 12308 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c26351720c0f48109cd9a0bb30c62f84: Sending DeleteTablet for 3 replicas of tablet 61707cf1fa1a418ea716d255998afde3
I20260929 10:54:40.969149 12587 tablet_service.cc:1514] Processing DeleteTablet for tablet 61707cf1fa1a418ea716d255998afde3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:54:40 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:51546
I20260929 10:54:40.969177 12447 tablet_service.cc:1514] Processing DeleteTablet for tablet 61707cf1fa1a418ea716d255998afde3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:54:40 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:36640
I20260929 10:54:40.969377 12727 tablet_service.cc:1514] Processing DeleteTablet for tablet 61707cf1fa1a418ea716d255998afde3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:54:40 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:55476
I20260929 10:54:40.969399 12877 tablet_replica.cc:330] T 61707cf1fa1a418ea716d255998afde3 P 91e8b37d2bcc4041a11630f9f303607d: stopping tablet replica
I20260929 10:54:40.969498 12877 raft_consensus.cc:2229] T 61707cf1fa1a418ea716d255998afde3 P 91e8b37d2bcc4041a11630f9f303607d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:54:40.969547 12877 raft_consensus.cc:2258] T 61707cf1fa1a418ea716d255998afde3 P 91e8b37d2bcc4041a11630f9f303607d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:54:40.969771 12877 ts_tablet_manager.cc:1905] T 61707cf1fa1a418ea716d255998afde3 P 91e8b37d2bcc4041a11630f9f303607d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:54:40.971560 12878 tablet_replica.cc:330] T 61707cf1fa1a418ea716d255998afde3 P 8f22daec740b421ab09c01da42631470: stopping tablet replica
I20260929 10:54:40.971642 12877 ts_tablet_manager.cc:1918] T 61707cf1fa1a418ea716d255998afde3 P 91e8b37d2bcc4041a11630f9f303607d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 10:54:40.971674 12878 raft_consensus.cc:2229] T 61707cf1fa1a418ea716d255998afde3 P 8f22daec740b421ab09c01da42631470 [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:54:40.971724 12877 log.cc:1196] T 61707cf1fa1a418ea716d255998afde3 P 91e8b37d2bcc4041a11630f9f303607d: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-0/wal/wals/61707cf1fa1a418ea716d255998afde3
I20260929 10:54:40.971758 12878 raft_consensus.cc:2258] T 61707cf1fa1a418ea716d255998afde3 P 8f22daec740b421ab09c01da42631470 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:54:40.971961 12877 ts_tablet_manager.cc:1939] T 61707cf1fa1a418ea716d255998afde3 P 91e8b37d2bcc4041a11630f9f303607d: Deleting consensus metadata
I20260929 10:54:40.972013 12878 ts_tablet_manager.cc:1905] T 61707cf1fa1a418ea716d255998afde3 P 8f22daec740b421ab09c01da42631470: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:54:40.972275 12295 catalog_manager.cc:4841] TS 91e8b37d2bcc4041a11630f9f303607d (127.10.238.193:40549): tablet 61707cf1fa1a418ea716d255998afde3 (table test-table [id=be2e09ee00f74a08bb056227cc8db1b4]) successfully deleted
I20260929 10:54:40.973363 12879 tablet_replica.cc:330] T 61707cf1fa1a418ea716d255998afde3 P 9bcd53f3752d4d80bcfcbba0c194d9a9: stopping tablet replica
I20260929 10:54:40.973439 12879 raft_consensus.cc:2229] T 61707cf1fa1a418ea716d255998afde3 P 9bcd53f3752d4d80bcfcbba0c194d9a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:54:40.973485 12879 raft_consensus.cc:2258] T 61707cf1fa1a418ea716d255998afde3 P 9bcd53f3752d4d80bcfcbba0c194d9a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:54:40.973703 12878 ts_tablet_manager.cc:1918] T 61707cf1fa1a418ea716d255998afde3 P 8f22daec740b421ab09c01da42631470: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 10:54:40.973718 12879 ts_tablet_manager.cc:1905] T 61707cf1fa1a418ea716d255998afde3 P 9bcd53f3752d4d80bcfcbba0c194d9a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:54:40.973769 12878 log.cc:1196] T 61707cf1fa1a418ea716d255998afde3 P 8f22daec740b421ab09c01da42631470: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-1/wal/wals/61707cf1fa1a418ea716d255998afde3
I20260929 10:54:40.974025 12878 ts_tablet_manager.cc:1939] T 61707cf1fa1a418ea716d255998afde3 P 8f22daec740b421ab09c01da42631470: Deleting consensus metadata
I20260929 10:54:40.974376 12297 catalog_manager.cc:4841] TS 8f22daec740b421ab09c01da42631470 (127.10.238.194:40521): tablet 61707cf1fa1a418ea716d255998afde3 (table test-table [id=be2e09ee00f74a08bb056227cc8db1b4]) successfully deleted
I20260929 10:54:40.975256 12879 ts_tablet_manager.cc:1918] T 61707cf1fa1a418ea716d255998afde3 P 9bcd53f3752d4d80bcfcbba0c194d9a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 10:54:40.975353 12879 log.cc:1196] T 61707cf1fa1a418ea716d255998afde3 P 9bcd53f3752d4d80bcfcbba0c194d9a9: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790679274510869-11195-0/minicluster-data/ts-2/wal/wals/61707cf1fa1a418ea716d255998afde3
I20260929 10:54:40.975590 12879 ts_tablet_manager.cc:1939] T 61707cf1fa1a418ea716d255998afde3 P 9bcd53f3752d4d80bcfcbba0c194d9a9: Deleting consensus metadata
I20260929 10:54:40.975929 12297 catalog_manager.cc:4841] TS 9bcd53f3752d4d80bcfcbba0c194d9a9 (127.10.238.195:39331): tablet 61707cf1fa1a418ea716d255998afde3 (table test-table [id=be2e09ee00f74a08bb056227cc8db1b4]) successfully deleted
Sep 29 10:54:40 dist-test-slave-8nrt krb5kdc[12265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679280, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.238.193@KRBTEST.COM
I20260929 10:54:40.983264 12809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:40.970385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36664 (local address 127.10.238.193:40549)
0929 10:54:40.970428 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:40.970430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:40.975234 (+ 4804us) server_negotiation.cc:407] Connection header received
0929 10:54:40.975302 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:40.975303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:40.975313 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:40.975359 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:40.976092 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.976590 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.977338 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.977545 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.977634 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:40.980363 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:40.980373 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:40.980373 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:40.980380 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:40.982045 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:40.982453 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:40.982456 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:40.982457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:40.982514 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:40.982708 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:40.982709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:40.982710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:40.982858 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:40.983004 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:40.983066 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 10:54:40.983145 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260929 10:54:40.983553 12507 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36664
I20260929 10:54:40.990772 12359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:40.984473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38638 (local address 127.10.238.254:41855)
0929 10:54:40.984517 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:40.984519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:40.984652 (+ 133us) server_negotiation.cc:407] Connection header received
0929 10:54:40.984726 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:40.984726 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:40.984737 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:40.984783 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:40.985318 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.985789 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.986583 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:40.986840 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:40.986897 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:40.988191 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:40.988192 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:40.988193 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:40.988201 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:40.989645 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:40.989996 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:40.989997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:40.989998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:40.990056 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:40.990260 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:40.990261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:40.990261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:40.990394 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:40.990498 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:40.990593 (+ 95us) server_negotiation.cc:299] Negotiation successful
0929 10:54:40.990667 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260929 10:54:40.990971 12308 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38638
I20260929 10:54:40.991616 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 12382
I20260929 10:54:40.999323 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 12522
I20260929 10:54:41.005245 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 12662
I20260929 10:54:41.009985 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 12278
2026-09-29T10:54:41Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2644 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:54:41 dist-test-slave-8nrt krb5kdc[12886](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:54:41 dist-test-slave-8nrt krb5kdc[12886](info): set up 2 sockets
Sep 29 10:54:41 dist-test-slave-8nrt krb5kdc[12886](info): commencing operation
krb5kdc: starting...
W20260929 10:54:42.056870 11195 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Sep 29 10:54:42 dist-test-slave-8nrt krb5kdc[12886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:54:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:54:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
I20260929 10:54:42.169907 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:41661
--webserver_interface=127.10.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:42485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:41661
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:54:42.251478 12899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:54:42.251686 12899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:54:42.251710 12899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:54:42.252899 12899 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:54:42.252942 12899 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 10:54:42.252959 12899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:54:42.252974 12899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:54:42.252987 12899 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:54:42.253001 12899 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:54:42.254398 12899 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:42485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:41661
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:41661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:54:42.254577 12899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:54:42.254755 12899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:54:42.259781 12907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:42.259729 12904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:42.259743 12905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:42.259824 12899 server_base.cc:1034] running on GCE node
I20260929 10:54:42.260147 12899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:54:42.260473 12899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:54:42.261621 12899 hybrid_clock.cc:648] HybridClock initialized: now 1790679282261600 us; error 37 us; skew 500 ppm
Sep 29 10:54:42 dist-test-slave-8nrt krb5kdc[12886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679282, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:54:42.264098 12899 init.cc:377] Logged in from keytab as kudu/127.10.238.254@KRBTEST.COM (short username kudu)
I20260929 10:54:42.264545 12899 webserver.cc:460] Webserver started at http://127.10.238.254:33161/ using document root <none> and password file <none>
I20260929 10:54:42.264742 12899 fs_manager.cc:359] Metadata directory not provided
I20260929 10:54:42.264784 12899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:54:42.264868 12899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:54:42.265904 12899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/master-0/data/instance:
uuid: "a789ec90b43c4dfe890712496bbac12f"
format_stamp: "Formatted at 2026-09-29 10:54:42 on dist-test-slave-8nrt"
server_key: "9c837eadedd4ce59778fc514a806b368"
server_key_iv: "600ce1a8b82797e8068b8e91d3e91083"
server_key_version: "encryptionkey@0"
I20260929 10:54:42.266202 12899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/master-0/wal/instance:
uuid: "a789ec90b43c4dfe890712496bbac12f"
format_stamp: "Formatted at 2026-09-29 10:54:42 on dist-test-slave-8nrt"
server_key: "9c837eadedd4ce59778fc514a806b368"
server_key_iv: "600ce1a8b82797e8068b8e91d3e91083"
server_key_version: "encryptionkey@0"
I20260929 10:54:42.267436 12899 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 10:54:42.268105 12914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:42.268265 12899 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 10:54:42.268366 12899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "a789ec90b43c4dfe890712496bbac12f"
format_stamp: "Formatted at 2026-09-29 10:54:42 on dist-test-slave-8nrt"
server_key: "9c837eadedd4ce59778fc514a806b368"
server_key_iv: "600ce1a8b82797e8068b8e91d3e91083"
server_key_version: "encryptionkey@0"
I20260929 10:54:42.268446 12899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:54:42.282253 12899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:54:42.284338 12899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:54:42.284498 12899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:54:42.289192 12899 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.254:41661
I20260929 10:54:42.289281 12976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.254:41661 every 8 connection(s)
I20260929 10:54:42.289593 12899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:54:42.290186 12977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:54:42.292428 12977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a789ec90b43c4dfe890712496bbac12f: Bootstrap starting.
I20260929 10:54:42.293056 12977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a789ec90b43c4dfe890712496bbac12f: Neither blocks nor log segments found. Creating new log.
I20260929 10:54:42.293427 12977 log.cc:824] T 00000000000000000000000000000000 P a789ec90b43c4dfe890712496bbac12f: Log is configured to *not* fsync() on all Append() calls
I20260929 10:54:42.294191 12977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a789ec90b43c4dfe890712496bbac12f: No bootstrap required, opened a new log
I20260929 10:54:42.294317 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 12899
I20260929 10:54:42.294406 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/master-0/wal/instance
I20260929 10:54:42.294602 11195 external_mini_cluster.cc:1428] Setting key b6a95487c7fee4735da5ef3e822c9942
I20260929 10:54:42.296800 12977 raft_consensus.cc:348] T 00000000000000000000000000000000 P a789ec90b43c4dfe890712496bbac12f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a789ec90b43c4dfe890712496bbac12f" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 41661 } }
I20260929 10:54:42.296977 12977 raft_consensus.cc:374] T 00000000000000000000000000000000 P a789ec90b43c4dfe890712496bbac12f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:54:42.297049 12977 raft_consensus.cc:729] T 00000000000000000000000000000000 P a789ec90b43c4dfe890712496bbac12f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a789ec90b43c4dfe890712496bbac12f, State: Initialized, Role: FOLLOWER
I20260929 10:54:42.297206 12977 consensus_queue.cc:260] T 00000000000000000000000000000000 P a789ec90b43c4dfe890712496bbac12f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a789ec90b43c4dfe890712496bbac12f" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 41661 } }
I20260929 10:54:42.297358 12977 raft_consensus.cc:388] T 00000000000000000000000000000000 P a789ec90b43c4dfe890712496bbac12f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:54:42.297410 12977 raft_consensus.cc:482] T 00000000000000000000000000000000 P a789ec90b43c4dfe890712496bbac12f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:54:42.297468 12977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a789ec90b43c4dfe890712496bbac12f [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:54:42.298084 12977 raft_consensus.cc:504] T 00000000000000000000000000000000 P a789ec90b43c4dfe890712496bbac12f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a789ec90b43c4dfe890712496bbac12f" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 41661 } }
I20260929 10:54:42.298221 12977 leader_election.cc:302] T 00000000000000000000000000000000 P a789ec90b43c4dfe890712496bbac12f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a789ec90b43c4dfe890712496bbac12f; no voters:
I20260929 10:54:42.298424 12977 leader_election.cc:288] T 00000000000000000000000000000000 P a789ec90b43c4dfe890712496bbac12f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:54:42.298461 12982 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a789ec90b43c4dfe890712496bbac12f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:54:42.298614 12982 raft_consensus.cc:686] T 00000000000000000000000000000000 P a789ec90b43c4dfe890712496bbac12f [term 1 LEADER]: Becoming Leader. State: Replica: a789ec90b43c4dfe890712496bbac12f, State: Running, Role: LEADER
I20260929 10:54:42.298691 12977 sys_catalog.cc:564] T 00000000000000000000000000000000 P a789ec90b43c4dfe890712496bbac12f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:54:42.298753 12982 consensus_queue.cc:237] T 00000000000000000000000000000000 P a789ec90b43c4dfe890712496bbac12f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a789ec90b43c4dfe890712496bbac12f" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 41661 } }
I20260929 10:54:42.299078 12982 sys_catalog.cc:455] T 00000000000000000000000000000000 P a789ec90b43c4dfe890712496bbac12f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a789ec90b43c4dfe890712496bbac12f. Latest consensus state: current_term: 1 leader_uuid: "a789ec90b43c4dfe890712496bbac12f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a789ec90b43c4dfe890712496bbac12f" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 41661 } } }
I20260929 10:54:42.299152 12982 sys_catalog.cc:458] T 00000000000000000000000000000000 P a789ec90b43c4dfe890712496bbac12f [sys.catalog]: This master's current role is: LEADER
I20260929 10:54:42.299391 12988 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:54:42.299909 12988 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:54:42.300369 12983 sys_catalog.cc:455] T 00000000000000000000000000000000 P a789ec90b43c4dfe890712496bbac12f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a789ec90b43c4dfe890712496bbac12f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a789ec90b43c4dfe890712496bbac12f" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 41661 } } }
I20260929 10:54:42.301182 12983 sys_catalog.cc:458] T 00000000000000000000000000000000 P a789ec90b43c4dfe890712496bbac12f [sys.catalog]: This master's current role is: LEADER
I20260929 10:54:42.301837 12988 catalog_manager.cc:1347] Generated new cluster ID: 6dd86bb16d10482e83996ae50199320e
I20260929 10:54:42.301895 12988 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 10:54:42 dist-test-slave-8nrt krb5kdc[12886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:54:42.308050 12981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:42.295107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38642 (local address 127.10.238.254:41661)
0929 10:54:42.295319 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:42.295334 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:42.295359 (+ 25us) server_negotiation.cc:407] Connection header received
0929 10:54:42.296776 (+ 1417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:42.296783 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:42.296831 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:42.296972 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:42.297757 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:42.298972 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:42.300168 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:42.300842 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:42.300953 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:42.303802 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:42.303806 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:42.303807 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:42.303818 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:42.306350 (+ 2532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:42.306773 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:42.306776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:42.306776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:42.306892 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:42.307127 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:42.307128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:42.307129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:42.307400 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:42.307585 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:42.307705 (+ 120us) server_negotiation.cc:299] Negotiation successful
0929 10:54:42.307862 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":110,"threads_started":1}
I20260929 10:54:42.312381 12988 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:54:42.313064 12988 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:54:42.323925 12988 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a789ec90b43c4dfe890712496bbac12f: Generated new TSK 0
I20260929 10:54:42.324111 12988 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.238.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab.
Entry for principal kudu/127.10.238.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab.
I20260929 10:54:42.352496 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.193:0
--local_ip_for_outbound_sockets=127.10.238.193
--webserver_interface=127.10.238.193
--webserver_port=0
--tserver_master_addrs=127.10.238.254:41661
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab
--principal=kudu/127.10.238.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:42485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:54:42.437645 13003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:54:42.437885 13003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:54:42.437929 13003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:54:42.439128 13003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:54:42.439191 13003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:54:42.439229 13003 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 10:54:42.439276 13003 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 10:54:42.439316 13003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.193
I20260929 10:54:42.440800 13003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:42485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab
--principal=kudu/127.10.238.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.238.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.238.254:41661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.193
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:54:42.441082 13003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:54:42.441378 13003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:54:42.446421 13008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:42.446439 13009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:42.446667 13003 server_base.cc:1034] running on GCE node
W20260929 10:54:42.446448 13011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:42.446952 13003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:54:42.447244 13003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:54:42.448405 13003 hybrid_clock.cc:648] HybridClock initialized: now 1790679282448389 us; error 46 us; skew 500 ppm
Sep 29 10:54:42 dist-test-slave-8nrt krb5kdc[12886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679282, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:54:42.451004 13003 init.cc:377] Logged in from keytab as kudu/127.10.238.193@KRBTEST.COM (short username kudu)
I20260929 10:54:42.451442 13003 webserver.cc:460] Webserver started at http://127.10.238.193:44427/ using document root <none> and password file <none>
I20260929 10:54:42.451664 13003 fs_manager.cc:359] Metadata directory not provided
I20260929 10:54:42.451742 13003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:54:42.451896 13003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:54:42.452948 13003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-0/data/instance:
uuid: "9ba1f180c87d40bdb61d85c762d7e342"
format_stamp: "Formatted at 2026-09-29 10:54:42 on dist-test-slave-8nrt"
server_key: "0dc5c37e72a32af9e42cab8c7917881c"
server_key_iv: "ef4215aff95287a64760712c6fc88b35"
server_key_version: "encryptionkey@0"
I20260929 10:54:42.453373 13003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-0/wal/instance:
uuid: "9ba1f180c87d40bdb61d85c762d7e342"
format_stamp: "Formatted at 2026-09-29 10:54:42 on dist-test-slave-8nrt"
server_key: "0dc5c37e72a32af9e42cab8c7917881c"
server_key_iv: "ef4215aff95287a64760712c6fc88b35"
server_key_version: "encryptionkey@0"
I20260929 10:54:42.454730 13003 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:54:42.455577 13018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:42.455777 13003 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:54:42.455878 13003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-0/wal
uuid: "9ba1f180c87d40bdb61d85c762d7e342"
format_stamp: "Formatted at 2026-09-29 10:54:42 on dist-test-slave-8nrt"
server_key: "0dc5c37e72a32af9e42cab8c7917881c"
server_key_iv: "ef4215aff95287a64760712c6fc88b35"
server_key_version: "encryptionkey@0"
I20260929 10:54:42.455963 13003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:54:42.466238 13003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:54:42.467978 13003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:54:42.468084 13003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:54:42.468511 13003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:54:42.468581 13003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:42.468626 13003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:54:42.468652 13003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:42.475471 13003 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.193:43443
I20260929 10:54:42.475570 13133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.193:43443 every 8 connection(s)
Sep 29 10:54:42 dist-test-slave-8nrt krb5kdc[12886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679282, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.193@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:54:42.475903 13003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
I20260929 10:54:42.477662 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 13003
I20260929 10:54:42.477777 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-0/wal/instance
I20260929 10:54:42.477993 11195 external_mini_cluster.cc:1428] Setting key 27efe954588900d3ce0681a6533da236
I20260929 10:54:42.479470 13027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:42.469037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:41661 (local address 127.10.238.193:36401)
0929 10:54:42.469197 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:42.469230 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:42.469771 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:42.470032 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:42.470035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:42.470405 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:54:42.470695 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:42.470703 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:42.471802 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:42.471803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:42.473124 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:42.473127 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:42.473660 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:42.473661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:42.473806 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:42.474348 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:54:42.474358 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:54:42.476236 (+ 1878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:54:42.477901 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:42.477909 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:42.477911 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:42.478215 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:42.478456 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:42.478457 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:42.478458 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:42.478624 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:42.478948 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:54:42.478954 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:54:42.479100 (+ 146us) client_negotiation.cc:769] Sending connection context
0929 10:54:42.479189 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 10:54:42.479314 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":93,"threads_started":1}
I20260929 10:54:42.479487 12981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:42.469050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:36401 (local address 127.10.238.254:41661)
0929 10:54:42.469092 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:42.469094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:42.469354 (+ 260us) server_negotiation.cc:407] Connection header received
0929 10:54:42.469882 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:42.469883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:42.469901 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:42.469964 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:42.470807 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:42.471682 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:42.473200 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:42.473560 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:42.473649 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:42.476369 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:42.476370 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:42.476371 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:42.476383 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:42.477701 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:42.478314 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:42.478315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:42.478316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:42.478380 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:42.478704 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:42.478705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:42.478705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:42.478859 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:42.478939 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:42.479239 (+ 300us) server_negotiation.cc:299] Negotiation successful
0929 10:54:42.479352 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 10:54:42.480520 13134 heartbeater.cc:347] Connected to a master server at 127.10.238.254:41661
I20260929 10:54:42.480655 13134 heartbeater.cc:464] Registering TS with master...
I20260929 10:54:42.480882 13134 heartbeater.cc:511] Master 127.10.238.254:41661 requested a full tablet report, sending...
I20260929 10:54:42.481516 12931 ts_manager.cc:194] Registered new tserver with Master: 9ba1f180c87d40bdb61d85c762d7e342 (127.10.238.193:43443)
I20260929 10:54:42.482453 12931 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.193@KRBTEST.COM'} at 127.10.238.193:36401
WARNING: no policy specified for kudu/127.10.238.194@KRBTEST.COM; defaulting to no policy
I20260929 10:54:42.487370 13141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:42.481559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:41661 (local address 127.10.238.193:58623)
0929 10:54:42.481693 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:42.481702 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:42.481793 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:42.481911 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:42.481911 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:42.482085 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:54:42.482137 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:42.482138 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:42.483015 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:42.483017 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:42.483609 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:42.483611 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:42.483997 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:42.483998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:42.484093 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:42.484545 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:54:42.484550 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:54:42.484834 (+ 284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:54:42.486530 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:42.486531 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:42.486532 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:42.486772 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:42.486949 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:42.486950 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:42.486950 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:42.487003 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:42.487154 (+ 151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:54:42.487155 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:54:42.487188 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 10:54:42.487236 (+ 48us) client_negotiation.cc:240] Negotiation successful
0929 10:54:42.487306 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":42,"thread_start_us":72,"threads_started":1}
I20260929 10:54:42.487380 12981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:42.481596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:58623 (local address 127.10.238.254:41661)
0929 10:54:42.481639 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:42.481641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:42.481770 (+ 129us) server_negotiation.cc:407] Connection header received
0929 10:54:42.481840 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:42.481841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:42.481852 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:42.481906 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:42.482214 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:42.482899 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:42.483705 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:42.483924 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:42.483999 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:42.484941 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:42.484942 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:42.484943 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:42.484949 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:42.486423 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:42.486840 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:42.486841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:42.486841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:42.486889 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:42.487040 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:42.487041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:42.487041 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:42.487120 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:42.487163 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:42.487249 (+ 86us) server_negotiation.cc:299] Negotiation successful
0929 10:54:42.487325 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab.
Entry for principal kudu/127.10.238.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab.
I20260929 10:54:42.503846 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.194:0
--local_ip_for_outbound_sockets=127.10.238.194
--webserver_interface=127.10.238.194
--webserver_port=0
--tserver_master_addrs=127.10.238.254:41661
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab
--principal=kudu/127.10.238.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:42485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:54:42.584455 13143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:54:42.584647 13143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:54:42.584671 13143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:54:42.585841 13143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:54:42.585882 13143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:54:42.585903 13143 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 10:54:42.585915 13143 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 10:54:42.585973 13143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.194
I20260929 10:54:42.587391 13143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:42485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab
--principal=kudu/127.10.238.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.238.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.238.254:41661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.194
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:54:42.587646 13143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:54:42.587870 13143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:54:42.592897 13149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:42.592877 13148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:42.592876 13151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:42.593144 13143 server_base.cc:1034] running on GCE node
I20260929 10:54:42.593403 13143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:54:42.593636 13143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:54:42.594810 13143 hybrid_clock.cc:648] HybridClock initialized: now 1790679282594781 us; error 40 us; skew 500 ppm
Sep 29 10:54:42 dist-test-slave-8nrt krb5kdc[12886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679282, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:54:42.597330 13143 init.cc:377] Logged in from keytab as kudu/127.10.238.194@KRBTEST.COM (short username kudu)
I20260929 10:54:42.597764 13143 webserver.cc:460] Webserver started at http://127.10.238.194:43143/ using document root <none> and password file <none>
I20260929 10:54:42.597940 13143 fs_manager.cc:359] Metadata directory not provided
I20260929 10:54:42.597980 13143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:54:42.598062 13143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:54:42.599153 13143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-1/data/instance:
uuid: "f2a14a2d305643c195076b4d55901271"
format_stamp: "Formatted at 2026-09-29 10:54:42 on dist-test-slave-8nrt"
server_key: "13f6344d9d370336d2d82d5ec6b8c8f4"
server_key_iv: "054374ac44355230b120cab986ebf17a"
server_key_version: "encryptionkey@0"
I20260929 10:54:42.599462 13143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-1/wal/instance:
uuid: "f2a14a2d305643c195076b4d55901271"
format_stamp: "Formatted at 2026-09-29 10:54:42 on dist-test-slave-8nrt"
server_key: "13f6344d9d370336d2d82d5ec6b8c8f4"
server_key_iv: "054374ac44355230b120cab986ebf17a"
server_key_version: "encryptionkey@0"
I20260929 10:54:42.600786 13143 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 10:54:42.601462 13158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:42.601635 13143 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:54:42.601727 13143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-1/wal
uuid: "f2a14a2d305643c195076b4d55901271"
format_stamp: "Formatted at 2026-09-29 10:54:42 on dist-test-slave-8nrt"
server_key: "13f6344d9d370336d2d82d5ec6b8c8f4"
server_key_iv: "054374ac44355230b120cab986ebf17a"
server_key_version: "encryptionkey@0"
I20260929 10:54:42.601802 13143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:54:42.612097 13143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:54:42.614307 13143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:54:42.614467 13143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:54:42.614941 13143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:54:42.614981 13143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:42.615003 13143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:54:42.615047 13143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:42.621886 13143 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.194:38251
I20260929 10:54:42.621950 13273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.194:38251 every 8 connection(s)
Sep 29 10:54:42 dist-test-slave-8nrt krb5kdc[12886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679282, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.194@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:54:42.622265 13143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
I20260929 10:54:42.624930 12981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:42.615442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:56205 (local address 127.10.238.254:41661)
0929 10:54:42.615522 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:42.615525 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:42.615722 (+ 197us) server_negotiation.cc:407] Connection header received
0929 10:54:42.616159 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:42.616161 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:42.616173 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:42.616257 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:42.617074 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:42.617963 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:42.619556 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:42.619909 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:42.619992 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:42.622726 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:42.622727 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:42.622727 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:42.622734 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:42.623833 (+ 1099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:42.624185 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:42.624185 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:42.624186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:42.624248 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:42.624453 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:42.624453 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:42.624454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:42.624551 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:42.624600 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:42.624812 (+ 212us) server_negotiation.cc:299] Negotiation successful
0929 10:54:42.624870 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 10:54:42.625093 13167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:42.615369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:41661 (local address 127.10.238.194:56205)
0929 10:54:42.615610 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:42.615636 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:42.616068 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:42.616303 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:42.616306 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:42.616687 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:54:42.616965 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:42.616971 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:42.618055 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:42.618056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:42.619454 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:42.619457 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:42.620009 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:42.620010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:42.620148 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:42.620672 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:54:42.620682 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:54:42.622625 (+ 1943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:54:42.623901 (+ 1276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:42.623907 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:42.623909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:42.624118 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:42.624306 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:42.624307 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:42.624307 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:42.624399 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:42.624603 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:54:42.624607 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:54:42.624708 (+ 101us) client_negotiation.cc:769] Sending connection context
0929 10:54:42.624796 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 10:54:42.624905 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":82,"threads_started":1}
I20260929 10:54:42.625597 13274 heartbeater.cc:347] Connected to a master server at 127.10.238.254:41661
I20260929 10:54:42.625702 13274 heartbeater.cc:464] Registering TS with master...
I20260929 10:54:42.625877 13274 heartbeater.cc:511] Master 127.10.238.254:41661 requested a full tablet report, sending...
I20260929 10:54:42.626303 12930 ts_manager.cc:194] Registered new tserver with Master: f2a14a2d305643c195076b4d55901271 (127.10.238.194:38251)
I20260929 10:54:42.626807 12930 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.194@KRBTEST.COM'} at 127.10.238.194:56205
I20260929 10:54:42.628055 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 13143
I20260929 10:54:42.628134 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-1/wal/instance
I20260929 10:54:42.628337 11195 external_mini_cluster.cc:1428] Setting key 39dc1e67b71d291cf8f20774ec92e2de
I20260929 10:54:42.632310 13280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:42.626543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:41661 (local address 127.10.238.194:38001)
0929 10:54:42.626670 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:42.626680 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:42.626736 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:42.626854 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:42.626855 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:42.627045 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:54:42.627108 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:42.627110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:42.627892 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:42.627892 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:42.628510 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:42.628512 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:42.628905 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:42.628906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:42.628981 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:42.629433 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:54:42.629437 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:54:42.629698 (+ 261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:54:42.631239 (+ 1541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:42.631240 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:42.631240 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:42.631495 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:42.631689 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:42.631690 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:42.631690 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:42.631737 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:42.632066 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:54:42.632067 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:54:42.632120 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 10:54:42.632171 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 10:54:42.632241 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":77,"threads_started":1}
I20260929 10:54:42.632421 12981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:42.626630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:38001 (local address 127.10.238.254:41661)
0929 10:54:42.626670 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:42.626671 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:42.626725 (+ 54us) server_negotiation.cc:407] Connection header received
0929 10:54:42.626797 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:42.626798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:42.626805 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:42.626848 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:42.627185 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:42.627808 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:42.628602 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:42.628832 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:42.628885 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:42.629776 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:42.629778 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:42.629778 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:42.629785 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:42.631158 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:42.631569 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:42.631570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:42.631570 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:42.631627 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:42.631797 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:42.631798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:42.631798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:42.631989 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:42.632068 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:42.632214 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 10:54:42.632293 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
WARNING: no policy specified for kudu/127.10.238.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab.
Entry for principal kudu/127.10.238.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab.
I20260929 10:54:42.653533 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.195:0
--local_ip_for_outbound_sockets=127.10.238.195
--webserver_interface=127.10.238.195
--webserver_port=0
--tserver_master_addrs=127.10.238.254:41661
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab
--principal=kudu/127.10.238.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:42485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:54:42.734432 13283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:54:42.734665 13283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:54:42.734705 13283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:54:42.735874 13283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:54:42.735948 13283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:54:42.735983 13283 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 10:54:42.736037 13283 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 10:54:42.736075 13283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.195
I20260929 10:54:42.737636 13283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:42485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab
--principal=kudu/127.10.238.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.238.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.238.254:41661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.195
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:54:42.737931 13283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:54:42.738173 13283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:54:42.743496 13289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:42.743498 13291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:42.743503 13288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:42.743721 13283 server_base.cc:1034] running on GCE node
I20260929 10:54:42.743927 13283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:54:42.744185 13283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:54:42.745328 13283 hybrid_clock.cc:648] HybridClock initialized: now 1790679282745317 us; error 32 us; skew 500 ppm
Sep 29 10:54:42 dist-test-slave-8nrt krb5kdc[12886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679282, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:54:42.752628 13283 init.cc:377] Logged in from keytab as kudu/127.10.238.195@KRBTEST.COM (short username kudu)
I20260929 10:54:42.753086 13283 webserver.cc:460] Webserver started at http://127.10.238.195:32947/ using document root <none> and password file <none>
I20260929 10:54:42.753387 13283 fs_manager.cc:359] Metadata directory not provided
I20260929 10:54:42.753459 13283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:54:42.753618 13283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:54:42.754680 13283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-2/data/instance:
uuid: "aeb660dae67845ffafb7f377e67753f3"
format_stamp: "Formatted at 2026-09-29 10:54:42 on dist-test-slave-8nrt"
server_key: "f2c7af741f77d199e30b43b962737ae3"
server_key_iv: "e5269812b7b5a1ad13a0483be40686ed"
server_key_version: "encryptionkey@0"
I20260929 10:54:42.755057 13283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-2/wal/instance:
uuid: "aeb660dae67845ffafb7f377e67753f3"
format_stamp: "Formatted at 2026-09-29 10:54:42 on dist-test-slave-8nrt"
server_key: "f2c7af741f77d199e30b43b962737ae3"
server_key_iv: "e5269812b7b5a1ad13a0483be40686ed"
server_key_version: "encryptionkey@0"
I20260929 10:54:42.756436 13283 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 10:54:42.757212 13298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:42.757421 13283 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 10:54:42.757514 13283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-2/wal
uuid: "aeb660dae67845ffafb7f377e67753f3"
format_stamp: "Formatted at 2026-09-29 10:54:42 on dist-test-slave-8nrt"
server_key: "f2c7af741f77d199e30b43b962737ae3"
server_key_iv: "e5269812b7b5a1ad13a0483be40686ed"
server_key_version: "encryptionkey@0"
I20260929 10:54:42.757609 13283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:54:42.774789 13283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:54:42.777026 13283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:54:42.777139 13283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:54:42.777658 13283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:54:42.777725 13283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:42.777767 13283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:54:42.777806 13283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:42.784745 13283 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.195:36733
I20260929 10:54:42.784821 13413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.195:36733 every 8 connection(s)
Sep 29 10:54:42 dist-test-slave-8nrt krb5kdc[12886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679282, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.195@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:54:42.785125 13283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
I20260929 10:54:42.787957 12981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:42.778113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:59281 (local address 127.10.238.254:41661)
0929 10:54:42.778186 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:42.778189 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:42.778389 (+ 200us) server_negotiation.cc:407] Connection header received
0929 10:54:42.778951 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:42.778952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:42.778965 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:42.779035 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:42.779985 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:42.780783 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:42.782464 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:42.782785 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:42.782849 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:42.785683 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:42.785685 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:42.785685 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:42.785693 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:42.786819 (+ 1126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:42.787174 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:42.787175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:42.787176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:42.787225 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:42.787434 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:42.787434 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:42.787435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:42.787532 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:42.787582 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:42.787831 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 10:54:42.787895 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 10:54:42.788139 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 13283
I20260929 10:54:42.788187 13307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:42.778095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:41661 (local address 127.10.238.195:59281)
0929 10:54:42.778279 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:42.778318 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:42.778834 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:42.779159 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:42.779164 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:42.779567 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:54:42.779877 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:42.779885 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:42.780898 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:42.780900 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:42.782355 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:42.782359 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:42.782870 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:42.782872 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:42.783052 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:42.783588 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:54:42.783599 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:54:42.785578 (+ 1979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:54:42.786891 (+ 1313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:42.786897 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:42.786899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:42.787108 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:42.787283 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:42.787284 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:42.787284 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:42.787379 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:42.787586 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:54:42.787590 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:54:42.787693 (+ 103us) client_negotiation.cc:769] Sending connection context
0929 10:54:42.787795 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 10:54:42.787909 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":118,"threads_started":1}
I20260929 10:54:42.788252 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-2/wal/instance
I20260929 10:54:42.788426 11195 external_mini_cluster.cc:1428] Setting key d8ed855e355dfbb3c9216993485950c9
I20260929 10:54:42.788689 13414 heartbeater.cc:347] Connected to a master server at 127.10.238.254:41661
I20260929 10:54:42.788800 13414 heartbeater.cc:464] Registering TS with master...
I20260929 10:54:42.789161 13414 heartbeater.cc:511] Master 127.10.238.254:41661 requested a full tablet report, sending...
I20260929 10:54:42.789553 12931 ts_manager.cc:194] Registered new tserver with Master: aeb660dae67845ffafb7f377e67753f3 (127.10.238.195:36733)
I20260929 10:54:42.790083 12931 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.195@KRBTEST.COM'} at 127.10.238.195:59281
I20260929 10:54:42.800060 13420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:42.789780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:41661 (local address 127.10.238.195:40485)
0929 10:54:42.789925 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:42.789939 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:42.789979 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:42.793314 (+ 3335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:42.793315 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:42.793546 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:54:42.793635 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:42.793637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:42.794541 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:42.794542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:42.795426 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:42.795429 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:42.795852 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:42.795853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:42.795931 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:42.796451 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:54:42.796457 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:54:42.796834 (+ 377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:54:42.798910 (+ 2076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:42.798911 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:42.798911 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:42.799189 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:42.799390 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:42.799391 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:42.799391 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:42.799449 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:42.799730 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:54:42.799731 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:54:42.799777 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 10:54:42.799865 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 10:54:42.799962 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":94,"threads_started":1}
I20260929 10:54:42.800051 12981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:42.793066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:40485 (local address 127.10.238.254:41661)
0929 10:54:42.793138 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:42.793140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:42.793148 (+ 8us) server_negotiation.cc:407] Connection header received
0929 10:54:42.793171 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:42.793172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:42.793180 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:42.793265 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:42.793736 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:42.794427 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:42.795518 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:42.795756 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:42.795850 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:42.796944 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:42.796945 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:42.796946 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:42.796952 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:42.798826 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:42.799258 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:42.799259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:42.799260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:42.799315 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:42.799533 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:42.799533 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:42.799534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:42.799638 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:42.799735 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:42.799890 (+ 155us) server_negotiation.cc:299] Negotiation successful
0929 10:54:42.799961 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260929 10:54:42.803902 11195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 10:54:42 dist-test-slave-8nrt krb5kdc[12886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 10:54:42 dist-test-slave-8nrt krb5kdc[12886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:54:42.826025 12981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:42.819278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38652 (local address 127.10.238.254:41661)
0929 10:54:42.819341 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:42.819343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:42.819351 (+ 8us) server_negotiation.cc:407] Connection header received
0929 10:54:42.819391 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:42.819392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:42.819400 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:42.819450 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:42.819969 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:42.820443 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:42.821121 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:42.821355 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:42.821406 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:42.823614 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:42.823615 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:42.823616 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:42.823622 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:42.824972 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:42.825313 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:42.825314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:42.825315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:42.825375 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:42.825611 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:42.825611 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:42.825611 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:42.825704 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:42.825776 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:42.825872 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 10:54:42.825941 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 10:54:42.827446 12931 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38652:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:54:42.828969 12931 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:54:42.835201 13431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:42.831170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.195:36733 (local address 127.0.0.1:33068)
0929 10:54:42.831282 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:42.831326 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:42.831391 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:42.831863 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:42.831866 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:42.831869 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:54:42.831929 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:42.831931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:42.832929 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:42.832930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:42.833961 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:42.833964 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:42.834882 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:42.834883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:42.834984 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:42.834998 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 10:54:42.835077 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 10:54:42.835131 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74,"thread_start_us":54,"threads_started":1}
I20260929 10:54:42.835245 13435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:42.831496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33068 (local address 127.10.238.195:36733)
0929 10:54:42.831673 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:42.831677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:42.831685 (+ 8us) server_negotiation.cc:407] Connection header received
0929 10:54:42.831715 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:42.831716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:42.831728 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:42.831854 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:54:42.832071 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:42.832838 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:42.834073 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:42.834806 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:42.834875 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:42.834931 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:42.835102 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 10:54:42.835148 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":87,"threads_started":1}
I20260929 10:54:42.835925 13433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:42.831365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:43443 (local address 127.0.0.1:40136)
0929 10:54:42.831769 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:42.831780 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:42.831853 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:42.832151 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:42.832152 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:42.832156 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:54:42.832219 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:42.832221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:42.833634 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:42.833636 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:42.834704 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:42.834706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:42.835694 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:42.835695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:42.835768 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:42.835771 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 10:54:42.835821 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 10:54:42.835863 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":67,"threads_started":1}
I20260929 10:54:42.836237 13436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:42.831755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40136 (local address 127.10.238.193:43443)
0929 10:54:42.831975 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:42.831980 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:42.831990 (+ 10us) server_negotiation.cc:407] Connection header received
0929 10:54:42.832031 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:42.832032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:42.832048 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:42.832415 (+ 367us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:54:42.832484 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:42.833541 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:42.834808 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:42.835595 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:42.835693 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:42.835745 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:42.836090 (+ 345us) server_negotiation.cc:299] Negotiation successful
0929 10:54:42.836129 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":109,"threads_started":1}
I20260929 10:54:42.836252 13348 tablet_service.cc:1467] Processing CreateTablet for tablet f6e363df7380414987ce798c1726adcd (DEFAULT_TABLE table=test-table [id=2cd107b6c4d64e41b7b45a5acd8e2b0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:54:42.836666 13348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6e363df7380414987ce798c1726adcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:54:42.837057 13432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:42.831174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.194:38251 (local address 127.0.0.1:34696)
0929 10:54:42.831319 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:42.831329 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:42.831391 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:42.832151 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:42.832152 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:42.832154 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:54:42.832299 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:42.832300 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:42.834815 (+ 2515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:42.834815 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:42.835857 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:42.835861 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:42.836863 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:42.836864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:42.836925 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:42.836929 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 10:54:42.836958 (+ 29us) client_negotiation.cc:240] Negotiation successful
0929 10:54:42.836992 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":89,"thread_start_us":74,"threads_started":1}
I20260929 10:54:42.837173 13068 tablet_service.cc:1467] Processing CreateTablet for tablet f6e363df7380414987ce798c1726adcd (DEFAULT_TABLE table=test-table [id=2cd107b6c4d64e41b7b45a5acd8e2b0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:54:42.837518 13434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:42.831375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34696 (local address 127.10.238.194:38251)
0929 10:54:42.831978 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:42.831981 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:42.831990 (+ 9us) server_negotiation.cc:407] Connection header received
0929 10:54:42.832030 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:42.832032 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:42.832048 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:42.832195 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:54:42.832371 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:42.833532 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:42.835951 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:42.836728 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:42.837237 (+ 509us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:42.837347 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:42.837377 (+ 30us) server_negotiation.cc:299] Negotiation successful
0929 10:54:42.837421 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":72,"threads_started":1}
I20260929 10:54:42.837590 13068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6e363df7380414987ce798c1726adcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:54:42.838120 13201 tablet_service.cc:1467] Processing CreateTablet for tablet f6e363df7380414987ce798c1726adcd (DEFAULT_TABLE table=test-table [id=2cd107b6c4d64e41b7b45a5acd8e2b0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:54:42.838444 13201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6e363df7380414987ce798c1726adcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:54:42.839761 13437 tablet_bootstrap.cc:492] T f6e363df7380414987ce798c1726adcd P aeb660dae67845ffafb7f377e67753f3: Bootstrap starting.
I20260929 10:54:42.840605 13437 tablet_bootstrap.cc:654] T f6e363df7380414987ce798c1726adcd P aeb660dae67845ffafb7f377e67753f3: Neither blocks nor log segments found. Creating new log.
I20260929 10:54:42.840924 13437 log.cc:824] T f6e363df7380414987ce798c1726adcd P aeb660dae67845ffafb7f377e67753f3: Log is configured to *not* fsync() on all Append() calls
I20260929 10:54:42.841321 13438 tablet_bootstrap.cc:492] T f6e363df7380414987ce798c1726adcd P 9ba1f180c87d40bdb61d85c762d7e342: Bootstrap starting.
I20260929 10:54:42.841866 13440 tablet_bootstrap.cc:492] T f6e363df7380414987ce798c1726adcd P f2a14a2d305643c195076b4d55901271: Bootstrap starting.
I20260929 10:54:42.841943 13437 tablet_bootstrap.cc:492] T f6e363df7380414987ce798c1726adcd P aeb660dae67845ffafb7f377e67753f3: No bootstrap required, opened a new log
I20260929 10:54:42.842061 13437 ts_tablet_manager.cc:1397] T f6e363df7380414987ce798c1726adcd P aeb660dae67845ffafb7f377e67753f3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 10:54:42.842201 13438 tablet_bootstrap.cc:654] T f6e363df7380414987ce798c1726adcd P 9ba1f180c87d40bdb61d85c762d7e342: Neither blocks nor log segments found. Creating new log.
I20260929 10:54:42.842566 13438 log.cc:824] T f6e363df7380414987ce798c1726adcd P 9ba1f180c87d40bdb61d85c762d7e342: Log is configured to *not* fsync() on all Append() calls
I20260929 10:54:42.842593 13440 tablet_bootstrap.cc:654] T f6e363df7380414987ce798c1726adcd P f2a14a2d305643c195076b4d55901271: Neither blocks nor log segments found. Creating new log.
I20260929 10:54:42.842908 13440 log.cc:824] T f6e363df7380414987ce798c1726adcd P f2a14a2d305643c195076b4d55901271: Log is configured to *not* fsync() on all Append() calls
I20260929 10:54:42.843379 13438 tablet_bootstrap.cc:492] T f6e363df7380414987ce798c1726adcd P 9ba1f180c87d40bdb61d85c762d7e342: No bootstrap required, opened a new log
I20260929 10:54:42.843448 13438 ts_tablet_manager.cc:1397] T f6e363df7380414987ce798c1726adcd P 9ba1f180c87d40bdb61d85c762d7e342: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 10:54:42.844408 13437 raft_consensus.cc:348] T f6e363df7380414987ce798c1726adcd P aeb660dae67845ffafb7f377e67753f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } }
I20260929 10:54:42.844576 13437 raft_consensus.cc:374] T f6e363df7380414987ce798c1726adcd P aeb660dae67845ffafb7f377e67753f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:54:42.844615 13437 raft_consensus.cc:729] T f6e363df7380414987ce798c1726adcd P aeb660dae67845ffafb7f377e67753f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeb660dae67845ffafb7f377e67753f3, State: Initialized, Role: FOLLOWER
I20260929 10:54:42.844743 13437 consensus_queue.cc:260] T f6e363df7380414987ce798c1726adcd P aeb660dae67845ffafb7f377e67753f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } }
I20260929 10:54:42.845001 13437 ts_tablet_manager.cc:1428] T f6e363df7380414987ce798c1726adcd P aeb660dae67845ffafb7f377e67753f3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 10:54:42.845047 13414 heartbeater.cc:503] Master 127.10.238.254:41661 was elected leader, sending a full tablet report...
I20260929 10:54:42.845546 13438 raft_consensus.cc:348] T f6e363df7380414987ce798c1726adcd P 9ba1f180c87d40bdb61d85c762d7e342 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } }
I20260929 10:54:42.845700 13438 raft_consensus.cc:374] T f6e363df7380414987ce798c1726adcd P 9ba1f180c87d40bdb61d85c762d7e342 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:54:42.845749 13438 raft_consensus.cc:729] T f6e363df7380414987ce798c1726adcd P 9ba1f180c87d40bdb61d85c762d7e342 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ba1f180c87d40bdb61d85c762d7e342, State: Initialized, Role: FOLLOWER
I20260929 10:54:42.845821 13440 tablet_bootstrap.cc:492] T f6e363df7380414987ce798c1726adcd P f2a14a2d305643c195076b4d55901271: No bootstrap required, opened a new log
I20260929 10:54:42.845881 13440 ts_tablet_manager.cc:1397] T f6e363df7380414987ce798c1726adcd P f2a14a2d305643c195076b4d55901271: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260929 10:54:42.845857 13438 consensus_queue.cc:260] T f6e363df7380414987ce798c1726adcd P 9ba1f180c87d40bdb61d85c762d7e342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } }
I20260929 10:54:42.846153 13134 heartbeater.cc:503] Master 127.10.238.254:41661 was elected leader, sending a full tablet report...
I20260929 10:54:42.846120 13438 ts_tablet_manager.cc:1428] T f6e363df7380414987ce798c1726adcd P 9ba1f180c87d40bdb61d85c762d7e342: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 10:54:42.847796 13440 raft_consensus.cc:348] T f6e363df7380414987ce798c1726adcd P f2a14a2d305643c195076b4d55901271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } }
I20260929 10:54:42.847915 13440 raft_consensus.cc:374] T f6e363df7380414987ce798c1726adcd P f2a14a2d305643c195076b4d55901271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:54:42.847935 13440 raft_consensus.cc:729] T f6e363df7380414987ce798c1726adcd P f2a14a2d305643c195076b4d55901271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2a14a2d305643c195076b4d55901271, State: Initialized, Role: FOLLOWER
I20260929 10:54:42.848006 13440 consensus_queue.cc:260] T f6e363df7380414987ce798c1726adcd P f2a14a2d305643c195076b4d55901271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } }
I20260929 10:54:42.848234 13440 ts_tablet_manager.cc:1428] T f6e363df7380414987ce798c1726adcd P f2a14a2d305643c195076b4d55901271: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 10:54:42.848330 13274 heartbeater.cc:503] Master 127.10.238.254:41661 was elected leader, sending a full tablet report...
W20260929 10:54:42.872926 13275 tablet.cc:2369] T f6e363df7380414987ce798c1726adcd P f2a14a2d305643c195076b4d55901271: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 10:54:42.976761 13135 tablet.cc:2369] T f6e363df7380414987ce798c1726adcd P 9ba1f180c87d40bdb61d85c762d7e342: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 10:54:43.035877 13415 tablet.cc:2369] T f6e363df7380414987ce798c1726adcd P aeb660dae67845ffafb7f377e67753f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:54:43.069802 13445 raft_consensus.cc:482] T f6e363df7380414987ce798c1726adcd P f2a14a2d305643c195076b4d55901271 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:54:43.069984 13445 raft_consensus.cc:504] T f6e363df7380414987ce798c1726adcd P f2a14a2d305643c195076b4d55901271 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } }
I20260929 10:54:43.070413 13445 leader_election.cc:288] T f6e363df7380414987ce798c1726adcd P f2a14a2d305643c195076b4d55901271 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aeb660dae67845ffafb7f377e67753f3 (127.10.238.195:36733), 9ba1f180c87d40bdb61d85c762d7e342 (127.10.238.193:43443)
I20260929 10:54:43.073542 13167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:43.070628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.195:36733 (local address 127.10.238.194:33407)
0929 10:54:43.070680 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:43.070692 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:43.070750 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:43.070916 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:43.070917 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:43.070919 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:54:43.071004 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:43.071006 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:43.071898 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:43.071899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:43.072697 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:43.072700 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:43.073246 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:43.073247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:43.073352 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:43.073355 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 10:54:43.073431 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 10:54:43.073472 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260929 10:54:43.073542 13435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:43.070679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:33407 (local address 127.10.238.195:36733)
0929 10:54:43.070744 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:43.070747 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:43.070758 (+ 11us) server_negotiation.cc:407] Connection header received
0929 10:54:43.070829 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:43.070830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:43.070843 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:43.070906 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:54:43.071119 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:43.071807 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:43.072795 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:43.073157 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:43.073246 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:43.073255 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:43.073440 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 10:54:43.073478 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 10:54:43.073711 13436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:43.070745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:55583 (local address 127.10.238.193:43443)
0929 10:54:43.070815 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:43.070837 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:43.071062 (+ 225us) server_negotiation.cc:407] Connection header received
0929 10:54:43.071142 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:43.071144 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:43.071155 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:43.071186 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:54:43.071409 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:43.071902 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:43.072903 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:43.073335 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:43.073426 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:43.073434 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:43.073604 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 10:54:43.073646 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 10:54:43.073729 13446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:43.070630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:43443 (local address 127.10.238.194:55583)
0929 10:54:43.070984 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:43.070993 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:43.071042 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:43.071266 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:43.071267 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:43.071269 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:54:43.071335 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:43.071337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:43.072003 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:43.072004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:43.072804 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:43.072807 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:43.073426 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:43.073427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:43.073514 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:43.073518 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 10:54:43.073591 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 10:54:43.073631 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":166,"thread_start_us":109,"threads_started":1}
I20260929 10:54:43.074146 13368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e363df7380414987ce798c1726adcd" candidate_uuid: "f2a14a2d305643c195076b4d55901271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeb660dae67845ffafb7f377e67753f3" is_pre_election: true
I20260929 10:54:43.074198 13088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e363df7380414987ce798c1726adcd" candidate_uuid: "f2a14a2d305643c195076b4d55901271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ba1f180c87d40bdb61d85c762d7e342" is_pre_election: true
I20260929 10:54:43.074328 13368 raft_consensus.cc:2454] T f6e363df7380414987ce798c1726adcd P aeb660dae67845ffafb7f377e67753f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2a14a2d305643c195076b4d55901271 in term 0.
I20260929 10:54:43.074371 13088 raft_consensus.cc:2454] T f6e363df7380414987ce798c1726adcd P 9ba1f180c87d40bdb61d85c762d7e342 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2a14a2d305643c195076b4d55901271 in term 0.
I20260929 10:54:43.074591 13161 leader_election.cc:302] T f6e363df7380414987ce798c1726adcd P f2a14a2d305643c195076b4d55901271 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ba1f180c87d40bdb61d85c762d7e342, f2a14a2d305643c195076b4d55901271; no voters:
I20260929 10:54:43.074752 13445 raft_consensus.cc:2781] T f6e363df7380414987ce798c1726adcd P f2a14a2d305643c195076b4d55901271 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:54:43.074820 13445 raft_consensus.cc:482] T f6e363df7380414987ce798c1726adcd P f2a14a2d305643c195076b4d55901271 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:54:43.074839 13445 raft_consensus.cc:3037] T f6e363df7380414987ce798c1726adcd P f2a14a2d305643c195076b4d55901271 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:54:43.075775 13445 raft_consensus.cc:504] T f6e363df7380414987ce798c1726adcd P f2a14a2d305643c195076b4d55901271 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } }
I20260929 10:54:43.075918 13445 leader_election.cc:288] T f6e363df7380414987ce798c1726adcd P f2a14a2d305643c195076b4d55901271 [CANDIDATE]: Term 1 election: Requested vote from peers aeb660dae67845ffafb7f377e67753f3 (127.10.238.195:36733), 9ba1f180c87d40bdb61d85c762d7e342 (127.10.238.193:43443)
I20260929 10:54:43.076052 13088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e363df7380414987ce798c1726adcd" candidate_uuid: "f2a14a2d305643c195076b4d55901271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ba1f180c87d40bdb61d85c762d7e342"
I20260929 10:54:43.076061 13368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e363df7380414987ce798c1726adcd" candidate_uuid: "f2a14a2d305643c195076b4d55901271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeb660dae67845ffafb7f377e67753f3"
I20260929 10:54:43.076139 13368 raft_consensus.cc:3037] T f6e363df7380414987ce798c1726adcd P aeb660dae67845ffafb7f377e67753f3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:54:43.076153 13088 raft_consensus.cc:3037] T f6e363df7380414987ce798c1726adcd P 9ba1f180c87d40bdb61d85c762d7e342 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:54:43.076823 13088 raft_consensus.cc:2454] T f6e363df7380414987ce798c1726adcd P 9ba1f180c87d40bdb61d85c762d7e342 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2a14a2d305643c195076b4d55901271 in term 1.
I20260929 10:54:43.076978 13368 raft_consensus.cc:2454] T f6e363df7380414987ce798c1726adcd P aeb660dae67845ffafb7f377e67753f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2a14a2d305643c195076b4d55901271 in term 1.
I20260929 10:54:43.077060 13161 leader_election.cc:302] T f6e363df7380414987ce798c1726adcd P f2a14a2d305643c195076b4d55901271 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ba1f180c87d40bdb61d85c762d7e342, f2a14a2d305643c195076b4d55901271; no voters:
I20260929 10:54:43.077142 13445 raft_consensus.cc:2781] T f6e363df7380414987ce798c1726adcd P f2a14a2d305643c195076b4d55901271 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:54:43.077332 13445 raft_consensus.cc:686] T f6e363df7380414987ce798c1726adcd P f2a14a2d305643c195076b4d55901271 [term 1 LEADER]: Becoming Leader. State: Replica: f2a14a2d305643c195076b4d55901271, State: Running, Role: LEADER
I20260929 10:54:43.077437 13445 consensus_queue.cc:237] T f6e363df7380414987ce798c1726adcd P f2a14a2d305643c195076b4d55901271 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } }
I20260929 10:54:43.078305 12931 catalog_manager.cc:5495] T f6e363df7380414987ce798c1726adcd P f2a14a2d305643c195076b4d55901271 reported cstate change: term changed from 0 to 1, leader changed from <none> to f2a14a2d305643c195076b4d55901271 (127.10.238.194). New cstate: current_term: 1 leader_uuid: "f2a14a2d305643c195076b4d55901271" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } health_report { overall_health: UNKNOWN } } }
Sep 29 10:54:43 dist-test-slave-8nrt krb5kdc[12886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679282, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:54:43.144937 12981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:43.138051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38666 (local address 127.10.238.254:41661)
0929 10:54:43.138162 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:43.138164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:43.138290 (+ 126us) server_negotiation.cc:407] Connection header received
0929 10:54:43.138375 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:43.138376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:43.138388 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:43.138460 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:43.138932 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:43.139418 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:43.140231 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:43.140407 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:43.140505 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:43.142532 (+ 2027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:43.142533 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:43.142533 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:43.142541 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:43.143697 (+ 1156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:43.144078 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:43.144079 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:43.144079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:43.144145 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:43.144340 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:43.144342 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:43.144342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:43.144451 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:43.144524 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:43.144692 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 10:54:43.144803 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":36}
I20260929 10:54:43.145001 13454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:43.137983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:41661 (local address 127.0.0.1:38666)
0929 10:54:43.138205 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:43.138222 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:43.138311 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:43.138463 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:43.138464 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:43.138799 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:54:43.138844 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:43.138847 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:43.139507 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:43.139508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:43.140135 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:43.140138 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:43.140504 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:43.140505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:43.140555 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:43.141069 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:54:43.141073 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:54:43.142442 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:54:43.143792 (+ 1350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:43.143800 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:43.143802 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:43.143989 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:43.144220 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:43.144221 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:43.144221 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:43.144279 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:43.144515 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:54:43.144522 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:54:43.144567 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 10:54:43.144668 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 10:54:43.144867 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":89,"thread_start_us":119,"threads_started":1}
I20260929 10:54:43.147013 12931 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:38666:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 10:54:43.147207 12931 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:54:43.148694 13068 tablet_service.cc:1467] Processing CreateTablet for tablet da5a3152959c4e9bac1d3242b43eab79 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e1e3a43b491e49df94ddd17f07d7061e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 10:54:43.148777 13348 tablet_service.cc:1467] Processing CreateTablet for tablet da5a3152959c4e9bac1d3242b43eab79 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e1e3a43b491e49df94ddd17f07d7061e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 10:54:43.148904 13068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da5a3152959c4e9bac1d3242b43eab79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:54:43.148962 13348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da5a3152959c4e9bac1d3242b43eab79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:54:43.148901 13201 tablet_service.cc:1467] Processing CreateTablet for tablet da5a3152959c4e9bac1d3242b43eab79 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e1e3a43b491e49df94ddd17f07d7061e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 10:54:43.149073 13201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da5a3152959c4e9bac1d3242b43eab79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:54:43.150179 13437 tablet_bootstrap.cc:492] T da5a3152959c4e9bac1d3242b43eab79 P aeb660dae67845ffafb7f377e67753f3: Bootstrap starting.
I20260929 10:54:43.150549 13440 tablet_bootstrap.cc:492] T da5a3152959c4e9bac1d3242b43eab79 P f2a14a2d305643c195076b4d55901271: Bootstrap starting.
I20260929 10:54:43.150785 13437 tablet_bootstrap.cc:654] T da5a3152959c4e9bac1d3242b43eab79 P aeb660dae67845ffafb7f377e67753f3: Neither blocks nor log segments found. Creating new log.
I20260929 10:54:43.150946 13440 tablet_bootstrap.cc:654] T da5a3152959c4e9bac1d3242b43eab79 P f2a14a2d305643c195076b4d55901271: Neither blocks nor log segments found. Creating new log.
I20260929 10:54:43.151310 13438 tablet_bootstrap.cc:492] T da5a3152959c4e9bac1d3242b43eab79 P 9ba1f180c87d40bdb61d85c762d7e342: Bootstrap starting.
I20260929 10:54:43.151540 13437 tablet_bootstrap.cc:492] T da5a3152959c4e9bac1d3242b43eab79 P aeb660dae67845ffafb7f377e67753f3: No bootstrap required, opened a new log
I20260929 10:54:43.151583 13440 tablet_bootstrap.cc:492] T da5a3152959c4e9bac1d3242b43eab79 P f2a14a2d305643c195076b4d55901271: No bootstrap required, opened a new log
I20260929 10:54:43.151587 13437 ts_tablet_manager.cc:1397] T da5a3152959c4e9bac1d3242b43eab79 P aeb660dae67845ffafb7f377e67753f3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 10:54:43.151621 13440 ts_tablet_manager.cc:1397] T da5a3152959c4e9bac1d3242b43eab79 P f2a14a2d305643c195076b4d55901271: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 10:54:43.151647 13438 tablet_bootstrap.cc:654] T da5a3152959c4e9bac1d3242b43eab79 P 9ba1f180c87d40bdb61d85c762d7e342: Neither blocks nor log segments found. Creating new log.
I20260929 10:54:43.151774 13437 raft_consensus.cc:348] T da5a3152959c4e9bac1d3242b43eab79 P aeb660dae67845ffafb7f377e67753f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } }
I20260929 10:54:43.151772 13440 raft_consensus.cc:348] T da5a3152959c4e9bac1d3242b43eab79 P f2a14a2d305643c195076b4d55901271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } }
I20260929 10:54:43.151839 13440 raft_consensus.cc:374] T da5a3152959c4e9bac1d3242b43eab79 P f2a14a2d305643c195076b4d55901271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:54:43.151839 13437 raft_consensus.cc:374] T da5a3152959c4e9bac1d3242b43eab79 P aeb660dae67845ffafb7f377e67753f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:54:43.151860 13437 raft_consensus.cc:729] T da5a3152959c4e9bac1d3242b43eab79 P aeb660dae67845ffafb7f377e67753f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeb660dae67845ffafb7f377e67753f3, State: Initialized, Role: FOLLOWER
I20260929 10:54:43.151860 13440 raft_consensus.cc:729] T da5a3152959c4e9bac1d3242b43eab79 P f2a14a2d305643c195076b4d55901271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2a14a2d305643c195076b4d55901271, State: Initialized, Role: FOLLOWER
I20260929 10:54:43.151904 13440 consensus_queue.cc:260] T da5a3152959c4e9bac1d3242b43eab79 P f2a14a2d305643c195076b4d55901271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } }
I20260929 10:54:43.151906 13437 consensus_queue.cc:260] T da5a3152959c4e9bac1d3242b43eab79 P aeb660dae67845ffafb7f377e67753f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } }
I20260929 10:54:43.151990 13440 ts_tablet_manager.cc:1428] T da5a3152959c4e9bac1d3242b43eab79 P f2a14a2d305643c195076b4d55901271: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 10:54:43.151990 13437 ts_tablet_manager.cc:1428] T da5a3152959c4e9bac1d3242b43eab79 P aeb660dae67845ffafb7f377e67753f3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:43.152067 13443 tablet_replica.cc:439] T da5a3152959c4e9bac1d3242b43eab79 P aeb660dae67845ffafb7f377e67753f3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } } }
I20260929 10:54:43.152137 13445 tablet_replica.cc:439] T da5a3152959c4e9bac1d3242b43eab79 P f2a14a2d305643c195076b4d55901271: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } } }
I20260929 10:54:43.152169 13443 tablet_replica.cc:442] T da5a3152959c4e9bac1d3242b43eab79 P aeb660dae67845ffafb7f377e67753f3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:54:43.152215 13445 tablet_replica.cc:442] T da5a3152959c4e9bac1d3242b43eab79 P f2a14a2d305643c195076b4d55901271: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:54:43.152626 13438 tablet_bootstrap.cc:492] T da5a3152959c4e9bac1d3242b43eab79 P 9ba1f180c87d40bdb61d85c762d7e342: No bootstrap required, opened a new log
I20260929 10:54:43.152675 13438 ts_tablet_manager.cc:1397] T da5a3152959c4e9bac1d3242b43eab79 P 9ba1f180c87d40bdb61d85c762d7e342: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 10:54:43.152830 13438 raft_consensus.cc:348] T da5a3152959c4e9bac1d3242b43eab79 P 9ba1f180c87d40bdb61d85c762d7e342 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } }
I20260929 10:54:43.152892 13438 raft_consensus.cc:374] T da5a3152959c4e9bac1d3242b43eab79 P 9ba1f180c87d40bdb61d85c762d7e342 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:54:43.152907 13438 raft_consensus.cc:729] T da5a3152959c4e9bac1d3242b43eab79 P 9ba1f180c87d40bdb61d85c762d7e342 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ba1f180c87d40bdb61d85c762d7e342, State: Initialized, Role: FOLLOWER
I20260929 10:54:43.152978 13438 consensus_queue.cc:260] T da5a3152959c4e9bac1d3242b43eab79 P 9ba1f180c87d40bdb61d85c762d7e342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } }
I20260929 10:54:43.153091 13438 ts_tablet_manager.cc:1428] T da5a3152959c4e9bac1d3242b43eab79 P 9ba1f180c87d40bdb61d85c762d7e342: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:43.153141 13444 tablet_replica.cc:439] T da5a3152959c4e9bac1d3242b43eab79 P 9ba1f180c87d40bdb61d85c762d7e342: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } } }
I20260929 10:54:43.153226 13444 tablet_replica.cc:442] T da5a3152959c4e9bac1d3242b43eab79 P 9ba1f180c87d40bdb61d85c762d7e342: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:54:43.174496 13444 raft_consensus.cc:482] T da5a3152959c4e9bac1d3242b43eab79 P 9ba1f180c87d40bdb61d85c762d7e342 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:54:43.174585 13444 raft_consensus.cc:504] T da5a3152959c4e9bac1d3242b43eab79 P 9ba1f180c87d40bdb61d85c762d7e342 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } }
I20260929 10:54:43.174949 13444 leader_election.cc:288] T da5a3152959c4e9bac1d3242b43eab79 P 9ba1f180c87d40bdb61d85c762d7e342 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2a14a2d305643c195076b4d55901271 (127.10.238.194:38251), aeb660dae67845ffafb7f377e67753f3 (127.10.238.195:36733)
I20260929 10:54:43.179083 13435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:43.175190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:42839 (local address 127.10.238.195:36733)
0929 10:54:43.175239 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:43.175242 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:43.175367 (+ 125us) server_negotiation.cc:407] Connection header received
0929 10:54:43.175486 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:43.175487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:43.175498 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:43.175553 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:54:43.175797 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:43.176565 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:43.178153 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:43.178702 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:43.178781 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:43.178791 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:43.178996 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 10:54:43.179028 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 10:54:43.179152 13458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:43.175127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.195:36733 (local address 127.10.238.193:42839)
0929 10:54:43.175264 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:43.175294 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:43.175388 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:43.175589 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:43.175591 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:43.175593 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:54:43.175720 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:43.175721 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:43.176651 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:43.176652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:43.178056 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:43.178060 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:43.178830 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:43.178831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:43.178908 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:43.178914 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 10:54:43.178983 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 10:54:43.179040 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":83,"thread_start_us":74,"threads_started":1}
I20260929 10:54:43.179409 13368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da5a3152959c4e9bac1d3242b43eab79" candidate_uuid: "9ba1f180c87d40bdb61d85c762d7e342" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeb660dae67845ffafb7f377e67753f3" is_pre_election: true
I20260929 10:54:43.179467 13459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:43.175127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.194:38251 (local address 127.10.238.193:46935)
0929 10:54:43.175398 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:43.175407 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:43.175463 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:43.175598 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:43.175598 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:43.175600 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:54:43.175703 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:43.175725 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:43.176650 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:43.176651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:43.178056 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:43.178060 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:43.179107 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:43.179109 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:43.179172 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:43.179176 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 10:54:43.179340 (+ 164us) client_negotiation.cc:240] Negotiation successful
0929 10:54:43.179380 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":102,"thread_start_us":70,"threads_started":1}
I20260929 10:54:43.179486 13434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:43.175203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:46935 (local address 127.10.238.194:38251)
0929 10:54:43.175248 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:43.175250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:43.175467 (+ 217us) server_negotiation.cc:407] Connection header received
0929 10:54:43.175520 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:43.175521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:43.175530 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:43.175584 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:54:43.175804 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:43.176557 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:43.178153 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:43.178702 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:43.178787 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:43.178795 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:43.179372 (+ 577us) server_negotiation.cc:299] Negotiation successful
0929 10:54:43.179425 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 10:54:43.179503 13368 raft_consensus.cc:2454] T da5a3152959c4e9bac1d3242b43eab79 P aeb660dae67845ffafb7f377e67753f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ba1f180c87d40bdb61d85c762d7e342 in term 0.
I20260929 10:54:43.179672 13020 leader_election.cc:302] T da5a3152959c4e9bac1d3242b43eab79 P 9ba1f180c87d40bdb61d85c762d7e342 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ba1f180c87d40bdb61d85c762d7e342, aeb660dae67845ffafb7f377e67753f3; no voters:
I20260929 10:54:43.179841 13228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da5a3152959c4e9bac1d3242b43eab79" candidate_uuid: "9ba1f180c87d40bdb61d85c762d7e342" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2a14a2d305643c195076b4d55901271" is_pre_election: true
I20260929 10:54:43.179948 13228 raft_consensus.cc:2454] T da5a3152959c4e9bac1d3242b43eab79 P f2a14a2d305643c195076b4d55901271 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ba1f180c87d40bdb61d85c762d7e342 in term 0.
I20260929 10:54:43.180018 13444 raft_consensus.cc:2781] T da5a3152959c4e9bac1d3242b43eab79 P 9ba1f180c87d40bdb61d85c762d7e342 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:54:43.180081 13444 raft_consensus.cc:482] T da5a3152959c4e9bac1d3242b43eab79 P 9ba1f180c87d40bdb61d85c762d7e342 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:54:43.180104 13444 raft_consensus.cc:3037] T da5a3152959c4e9bac1d3242b43eab79 P 9ba1f180c87d40bdb61d85c762d7e342 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:54:43.180850 13444 raft_consensus.cc:504] T da5a3152959c4e9bac1d3242b43eab79 P 9ba1f180c87d40bdb61d85c762d7e342 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } }
I20260929 10:54:43.180989 13444 leader_election.cc:288] T da5a3152959c4e9bac1d3242b43eab79 P 9ba1f180c87d40bdb61d85c762d7e342 [CANDIDATE]: Term 1 election: Requested vote from peers f2a14a2d305643c195076b4d55901271 (127.10.238.194:38251), aeb660dae67845ffafb7f377e67753f3 (127.10.238.195:36733)
I20260929 10:54:43.181130 13228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da5a3152959c4e9bac1d3242b43eab79" candidate_uuid: "9ba1f180c87d40bdb61d85c762d7e342" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2a14a2d305643c195076b4d55901271"
I20260929 10:54:43.181223 13228 raft_consensus.cc:3037] T da5a3152959c4e9bac1d3242b43eab79 P f2a14a2d305643c195076b4d55901271 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:54:43.181213 13368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da5a3152959c4e9bac1d3242b43eab79" candidate_uuid: "9ba1f180c87d40bdb61d85c762d7e342" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeb660dae67845ffafb7f377e67753f3"
I20260929 10:54:43.181331 13368 raft_consensus.cc:3037] T da5a3152959c4e9bac1d3242b43eab79 P aeb660dae67845ffafb7f377e67753f3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:54:43.181967 13228 raft_consensus.cc:2454] T da5a3152959c4e9bac1d3242b43eab79 P f2a14a2d305643c195076b4d55901271 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ba1f180c87d40bdb61d85c762d7e342 in term 1.
I20260929 10:54:43.182076 13368 raft_consensus.cc:2454] T da5a3152959c4e9bac1d3242b43eab79 P aeb660dae67845ffafb7f377e67753f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ba1f180c87d40bdb61d85c762d7e342 in term 1.
I20260929 10:54:43.182204 13021 leader_election.cc:302] T da5a3152959c4e9bac1d3242b43eab79 P 9ba1f180c87d40bdb61d85c762d7e342 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ba1f180c87d40bdb61d85c762d7e342, f2a14a2d305643c195076b4d55901271; no voters:
I20260929 10:54:43.182297 13444 raft_consensus.cc:2781] T da5a3152959c4e9bac1d3242b43eab79 P 9ba1f180c87d40bdb61d85c762d7e342 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:54:43.182427 13444 raft_consensus.cc:686] T da5a3152959c4e9bac1d3242b43eab79 P 9ba1f180c87d40bdb61d85c762d7e342 [term 1 LEADER]: Becoming Leader. State: Replica: 9ba1f180c87d40bdb61d85c762d7e342, State: Running, Role: LEADER
I20260929 10:54:43.182544 13444 consensus_queue.cc:237] T da5a3152959c4e9bac1d3242b43eab79 P 9ba1f180c87d40bdb61d85c762d7e342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } }
I20260929 10:54:43.182987 13460 tablet_replica.cc:439] T da5a3152959c4e9bac1d3242b43eab79 P 9ba1f180c87d40bdb61d85c762d7e342: TxnStatusTablet state changed. Reason: New leader 9ba1f180c87d40bdb61d85c762d7e342. Latest consensus state: current_term: 1 leader_uuid: "9ba1f180c87d40bdb61d85c762d7e342" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } } }
I20260929 10:54:43.183108 13460 tablet_replica.cc:442] T da5a3152959c4e9bac1d3242b43eab79 P 9ba1f180c87d40bdb61d85c762d7e342: This TxnStatusTablet replica's current role is: LEADER
I20260929 10:54:43.183344 13462 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 10:54:43.183455 13462 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 10:54:43.183358 12930 catalog_manager.cc:5495] T da5a3152959c4e9bac1d3242b43eab79 P 9ba1f180c87d40bdb61d85c762d7e342 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ba1f180c87d40bdb61d85c762d7e342 (127.10.238.193). New cstate: current_term: 1 leader_uuid: "9ba1f180c87d40bdb61d85c762d7e342" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } health_report { overall_health: UNKNOWN } } }
I20260929 10:54:43.190516 13436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:43.187445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40148 (local address 127.10.238.193:43443)
0929 10:54:43.187480 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:43.187482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:43.187532 (+ 50us) server_negotiation.cc:407] Connection header received
0929 10:54:43.187625 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:43.187626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:43.187641 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:43.187696 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:54:43.187856 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:43.188379 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:43.189529 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:43.189789 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:43.189916 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:43.190149 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:54:43.190274 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:54:43.190369 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:43.190428 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 10:54:43.190449 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 10:54:43.190555 13454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:43.187391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:43443 (local address 127.0.0.1:40148)
0929 10:54:43.187471 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:43.187482 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:43.187550 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:43.187733 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:43.187735 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:43.187736 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 10:54:43.187791 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:43.187794 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:43.188453 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:43.188454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:43.189442 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:43.189447 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:43.189917 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:43.189919 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:43.189993 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:43.190018 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 10:54:43.190348 (+ 330us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 10:54:43.190359 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 10:54:43.190422 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 10:54:43.190473 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 10:54:43.192698 13228 raft_consensus.cc:1261] T da5a3152959c4e9bac1d3242b43eab79 P f2a14a2d305643c195076b4d55901271 [term 1 FOLLOWER]: Refusing update from remote peer 9ba1f180c87d40bdb61d85c762d7e342: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:54:43.192731 13368 raft_consensus.cc:1261] T da5a3152959c4e9bac1d3242b43eab79 P aeb660dae67845ffafb7f377e67753f3 [term 1 FOLLOWER]: Refusing update from remote peer 9ba1f180c87d40bdb61d85c762d7e342: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:54:43.192986 13460 consensus_queue.cc:1035] T da5a3152959c4e9bac1d3242b43eab79 P 9ba1f180c87d40bdb61d85c762d7e342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:54:43.193156 13444 consensus_queue.cc:1035] T da5a3152959c4e9bac1d3242b43eab79 P 9ba1f180c87d40bdb61d85c762d7e342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:54:43.194310 13445 tablet_replica.cc:439] T da5a3152959c4e9bac1d3242b43eab79 P f2a14a2d305643c195076b4d55901271: TxnStatusTablet state changed. Reason: New leader 9ba1f180c87d40bdb61d85c762d7e342. Latest consensus state: current_term: 1 leader_uuid: "9ba1f180c87d40bdb61d85c762d7e342" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } } }
I20260929 10:54:43.194407 13445 tablet_replica.cc:442] T da5a3152959c4e9bac1d3242b43eab79 P f2a14a2d305643c195076b4d55901271: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:54:43.194413 13443 tablet_replica.cc:439] T da5a3152959c4e9bac1d3242b43eab79 P aeb660dae67845ffafb7f377e67753f3: TxnStatusTablet state changed. Reason: New leader 9ba1f180c87d40bdb61d85c762d7e342. Latest consensus state: current_term: 1 leader_uuid: "9ba1f180c87d40bdb61d85c762d7e342" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } } }
I20260929 10:54:43.194484 13443 tablet_replica.cc:442] T da5a3152959c4e9bac1d3242b43eab79 P aeb660dae67845ffafb7f377e67753f3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:54:43.195422 13444 tablet_replica.cc:439] T da5a3152959c4e9bac1d3242b43eab79 P 9ba1f180c87d40bdb61d85c762d7e342: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9ba1f180c87d40bdb61d85c762d7e342" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } } }
I20260929 10:54:43.195511 13444 tablet_replica.cc:442] T da5a3152959c4e9bac1d3242b43eab79 P 9ba1f180c87d40bdb61d85c762d7e342: This TxnStatusTablet replica's current role is: LEADER
I20260929 10:54:43.195689 13445 tablet_replica.cc:439] T da5a3152959c4e9bac1d3242b43eab79 P f2a14a2d305643c195076b4d55901271: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9ba1f180c87d40bdb61d85c762d7e342" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } } }
I20260929 10:54:43.195771 13445 tablet_replica.cc:442] T da5a3152959c4e9bac1d3242b43eab79 P f2a14a2d305643c195076b4d55901271: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:54:43.196106 13467 mvcc.cc:205] Tried to move back new op lower bound from 7334622343954841600 to 7334622343916331008. Current Snapshot: MvccSnapshot[applied={T|T < 7334622343954841600 or (T in {7334622343954841600})}]
I20260929 10:54:43.196285 13465 mvcc.cc:205] Tried to move back new op lower bound from 7334622343954841600 to 7334622343916331008. Current Snapshot: MvccSnapshot[applied={T|T < 7334622343954841600}]
I20260929 10:54:43.196408 13460 tablet_replica.cc:439] T da5a3152959c4e9bac1d3242b43eab79 P 9ba1f180c87d40bdb61d85c762d7e342: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9ba1f180c87d40bdb61d85c762d7e342" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } } }
I20260929 10:54:43.196499 13460 tablet_replica.cc:442] T da5a3152959c4e9bac1d3242b43eab79 P 9ba1f180c87d40bdb61d85c762d7e342: This TxnStatusTablet replica's current role is: LEADER
I20260929 10:54:43.197049 13443 tablet_replica.cc:439] T da5a3152959c4e9bac1d3242b43eab79 P aeb660dae67845ffafb7f377e67753f3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9ba1f180c87d40bdb61d85c762d7e342" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } } }
I20260929 10:54:43.197127 13466 mvcc.cc:205] Tried to move back new op lower bound from 7334622343954841600 to 7334622343916331008. Current Snapshot: MvccSnapshot[applied={T|T < 7334622343954841600}]
I20260929 10:54:43.197167 13443 tablet_replica.cc:442] T da5a3152959c4e9bac1d3242b43eab79 P aeb660dae67845ffafb7f377e67753f3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:54:43.201859 13434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:43.198904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34710 (local address 127.10.238.194:38251)
0929 10:54:43.198945 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:43.198946 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:43.198955 (+ 9us) server_negotiation.cc:407] Connection header received
0929 10:54:43.199034 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:43.199034 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:43.199052 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:43.199127 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:54:43.199283 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:43.200026 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:43.200985 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:43.201311 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:43.201379 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:43.201514 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:54:43.201624 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:54:43.201702 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:43.201770 (+ 68us) server_negotiation.cc:299] Negotiation successful
0929 10:54:43.201791 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 10:54:43.207113 13436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:43.204507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:41061 (local address 127.10.238.193:43443)
0929 10:54:43.204553 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:43.204555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:43.204645 (+ 90us) server_negotiation.cc:407] Connection header received
0929 10:54:43.204674 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:43.204675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:43.204685 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:43.204760 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:54:43.204852 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:43.205644 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:43.206357 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:43.206649 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:43.206724 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:43.206863 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:54:43.206895 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:54:43.206946 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:43.207029 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 10:54:43.207047 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 10:54:43.207111 13476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:43.204428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:43443 (local address 127.10.238.194:41061)
0929 10:54:43.204569 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:43.204579 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:43.204640 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:43.204760 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:43.204761 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:43.204762 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 10:54:43.204816 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:43.204816 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:43.205713 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:43.205713 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:43.206282 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:43.206283 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:43.206721 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:43.206721 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:43.206766 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:43.206778 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 10:54:43.206958 (+ 180us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 10:54:43.206960 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 10:54:43.207013 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 10:54:43.207044 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":76,"threads_started":1}
I20260929 10:54:43.209277 13368 raft_consensus.cc:1261] T f6e363df7380414987ce798c1726adcd P aeb660dae67845ffafb7f377e67753f3 [term 1 FOLLOWER]: Refusing update from remote peer f2a14a2d305643c195076b4d55901271: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:54:43.209359 13088 raft_consensus.cc:1261] T f6e363df7380414987ce798c1726adcd P 9ba1f180c87d40bdb61d85c762d7e342 [term 1 FOLLOWER]: Refusing update from remote peer f2a14a2d305643c195076b4d55901271: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:54:43.209627 13447 consensus_queue.cc:1035] T f6e363df7380414987ce798c1726adcd P f2a14a2d305643c195076b4d55901271 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:54:43.209765 13445 consensus_queue.cc:1035] T f6e363df7380414987ce798c1726adcd P f2a14a2d305643c195076b4d55901271 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:54:43.219348 13434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:43.215833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:43371 (local address 127.10.238.194:38251)
0929 10:54:43.215875 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:43.215877 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:43.216326 (+ 449us) server_negotiation.cc:407] Connection header received
0929 10:54:43.216435 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:43.216436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:43.216446 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:43.216495 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:54:43.216662 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:43.217312 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:43.218534 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:43.218834 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:43.218904 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:43.219101 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:54:43.219131 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:54:43.219175 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:43.219234 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 10:54:43.219251 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 10:54:43.219388 13482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:43.215791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.194:38251 (local address 127.10.238.193:43371)
0929 10:54:43.216237 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:43.216248 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:43.216370 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:43.216495 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:43.216496 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:43.216497 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 10:54:43.216563 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:43.216565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:43.217383 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:43.217385 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:43.218412 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:43.218415 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:43.218916 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:43.218917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:43.218997 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:43.219013 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 10:54:43.219189 (+ 176us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 10:54:43.219191 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 10:54:43.219234 (+ 43us) client_negotiation.cc:240] Negotiation successful
0929 10:54:43.219276 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":89,"threads_started":1}
I20260929 10:54:43.228025 12931 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:54:43.228137 12931 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:54:43.228888 12931 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a789ec90b43c4dfe890712496bbac12f: Sending DeleteTablet for 3 replicas of tablet f6e363df7380414987ce798c1726adcd
I20260929 10:54:43.229171 13201 tablet_service.cc:1514] Processing DeleteTablet for tablet f6e363df7380414987ce798c1726adcd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:54:43 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:34696
I20260929 10:54:43.229213 13068 tablet_service.cc:1514] Processing DeleteTablet for tablet f6e363df7380414987ce798c1726adcd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:54:43 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:40136
I20260929 10:54:43.229540 13348 tablet_service.cc:1514] Processing DeleteTablet for tablet f6e363df7380414987ce798c1726adcd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:54:43 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:33068
I20260929 10:54:43.229756 12931 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38652:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:54:43.229904 12931 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:54:43.230290 13487 tablet_replica.cc:330] T f6e363df7380414987ce798c1726adcd P 9ba1f180c87d40bdb61d85c762d7e342: stopping tablet replica
I20260929 10:54:43.230459 13487 raft_consensus.cc:2229] T f6e363df7380414987ce798c1726adcd P 9ba1f180c87d40bdb61d85c762d7e342 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:54:43.230566 13487 raft_consensus.cc:2258] T f6e363df7380414987ce798c1726adcd P 9ba1f180c87d40bdb61d85c762d7e342 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:54:43.231207 13348 tablet_service.cc:1467] Processing CreateTablet for tablet 722af37adf4c4a579563e562965f07b4 (DEFAULT_TABLE table=test-table [id=0536518b664e481e9c0367058b3da09e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:54:43.231230 13068 tablet_service.cc:1467] Processing CreateTablet for tablet 722af37adf4c4a579563e562965f07b4 (DEFAULT_TABLE table=test-table [id=0536518b664e481e9c0367058b3da09e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:54:43.231333 13348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 722af37adf4c4a579563e562965f07b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:54:43.231338 13068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 722af37adf4c4a579563e562965f07b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:54:43.231355 13487 ts_tablet_manager.cc:1905] T f6e363df7380414987ce798c1726adcd P 9ba1f180c87d40bdb61d85c762d7e342: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:54:43.232676 13437 tablet_bootstrap.cc:492] T 722af37adf4c4a579563e562965f07b4 P aeb660dae67845ffafb7f377e67753f3: Bootstrap starting.
I20260929 10:54:43.232734 13438 tablet_bootstrap.cc:492] T 722af37adf4c4a579563e562965f07b4 P 9ba1f180c87d40bdb61d85c762d7e342: Bootstrap starting.
I20260929 10:54:43.233067 13437 tablet_bootstrap.cc:654] T 722af37adf4c4a579563e562965f07b4 P aeb660dae67845ffafb7f377e67753f3: Neither blocks nor log segments found. Creating new log.
I20260929 10:54:43.233168 13438 tablet_bootstrap.cc:654] T 722af37adf4c4a579563e562965f07b4 P 9ba1f180c87d40bdb61d85c762d7e342: Neither blocks nor log segments found. Creating new log.
I20260929 10:54:43.233809 13201 tablet_service.cc:1467] Processing CreateTablet for tablet 722af37adf4c4a579563e562965f07b4 (DEFAULT_TABLE table=test-table [id=0536518b664e481e9c0367058b3da09e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:54:43.233919 13201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 722af37adf4c4a579563e562965f07b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:54:43.235253 13440 tablet_bootstrap.cc:492] T 722af37adf4c4a579563e562965f07b4 P f2a14a2d305643c195076b4d55901271: Bootstrap starting.
I20260929 10:54:43.235718 13440 tablet_bootstrap.cc:654] T 722af37adf4c4a579563e562965f07b4 P f2a14a2d305643c195076b4d55901271: Neither blocks nor log segments found. Creating new log.
I20260929 10:54:43.236469 13440 tablet_bootstrap.cc:492] T 722af37adf4c4a579563e562965f07b4 P f2a14a2d305643c195076b4d55901271: No bootstrap required, opened a new log
I20260929 10:54:43.236522 13440 ts_tablet_manager.cc:1397] T 722af37adf4c4a579563e562965f07b4 P f2a14a2d305643c195076b4d55901271: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 10:54:43.236657 13440 raft_consensus.cc:348] T 722af37adf4c4a579563e562965f07b4 P f2a14a2d305643c195076b4d55901271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } }
I20260929 10:54:43.236737 13440 raft_consensus.cc:374] T 722af37adf4c4a579563e562965f07b4 P f2a14a2d305643c195076b4d55901271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:54:43.236760 13440 raft_consensus.cc:729] T 722af37adf4c4a579563e562965f07b4 P f2a14a2d305643c195076b4d55901271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2a14a2d305643c195076b4d55901271, State: Initialized, Role: FOLLOWER
I20260929 10:54:43.236804 13440 consensus_queue.cc:260] T 722af37adf4c4a579563e562965f07b4 P f2a14a2d305643c195076b4d55901271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } }
I20260929 10:54:43.236892 13440 ts_tablet_manager.cc:1428] T 722af37adf4c4a579563e562965f07b4 P f2a14a2d305643c195076b4d55901271: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 10:54:43.233237 13487 ts_tablet_manager.cc:1918] T f6e363df7380414987ce798c1726adcd P 9ba1f180c87d40bdb61d85c762d7e342: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 10:54:43.237448 13487 log.cc:1196] T f6e363df7380414987ce798c1726adcd P 9ba1f180c87d40bdb61d85c762d7e342: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-0/wal/wals/f6e363df7380414987ce798c1726adcd
I20260929 10:54:43.237627 13437 tablet_bootstrap.cc:492] T 722af37adf4c4a579563e562965f07b4 P aeb660dae67845ffafb7f377e67753f3: No bootstrap required, opened a new log
I20260929 10:54:43.237627 13438 tablet_bootstrap.cc:492] T 722af37adf4c4a579563e562965f07b4 P 9ba1f180c87d40bdb61d85c762d7e342: No bootstrap required, opened a new log
I20260929 10:54:43.237674 13437 ts_tablet_manager.cc:1397] T 722af37adf4c4a579563e562965f07b4 P aeb660dae67845ffafb7f377e67753f3: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260929 10:54:43.237674 13438 ts_tablet_manager.cc:1397] T 722af37adf4c4a579563e562965f07b4 P 9ba1f180c87d40bdb61d85c762d7e342: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260929 10:54:43.237876 13487 ts_tablet_manager.cc:1939] T f6e363df7380414987ce798c1726adcd P 9ba1f180c87d40bdb61d85c762d7e342: Deleting consensus metadata
I20260929 10:54:43.237850 13437 raft_consensus.cc:348] T 722af37adf4c4a579563e562965f07b4 P aeb660dae67845ffafb7f377e67753f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } }
I20260929 10:54:43.237854 13438 raft_consensus.cc:348] T 722af37adf4c4a579563e562965f07b4 P 9ba1f180c87d40bdb61d85c762d7e342 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } }
I20260929 10:54:43.237922 13437 raft_consensus.cc:374] T 722af37adf4c4a579563e562965f07b4 P aeb660dae67845ffafb7f377e67753f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:54:43.237929 13438 raft_consensus.cc:374] T 722af37adf4c4a579563e562965f07b4 P 9ba1f180c87d40bdb61d85c762d7e342 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:54:43.237948 13437 raft_consensus.cc:729] T 722af37adf4c4a579563e562965f07b4 P aeb660dae67845ffafb7f377e67753f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeb660dae67845ffafb7f377e67753f3, State: Initialized, Role: FOLLOWER
I20260929 10:54:43.237952 13438 raft_consensus.cc:729] T 722af37adf4c4a579563e562965f07b4 P 9ba1f180c87d40bdb61d85c762d7e342 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ba1f180c87d40bdb61d85c762d7e342, State: Initialized, Role: FOLLOWER
I20260929 10:54:43.238000 13438 consensus_queue.cc:260] T 722af37adf4c4a579563e562965f07b4 P 9ba1f180c87d40bdb61d85c762d7e342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } }
I20260929 10:54:43.237999 13437 consensus_queue.cc:260] T 722af37adf4c4a579563e562965f07b4 P aeb660dae67845ffafb7f377e67753f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } }
I20260929 10:54:43.238085 13437 ts_tablet_manager.cc:1428] T 722af37adf4c4a579563e562965f07b4 P aeb660dae67845ffafb7f377e67753f3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:43.238099 13438 ts_tablet_manager.cc:1428] T 722af37adf4c4a579563e562965f07b4 P 9ba1f180c87d40bdb61d85c762d7e342: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260929 10:54:43.238345 12917 catalog_manager.cc:4841] TS 9ba1f180c87d40bdb61d85c762d7e342 (127.10.238.193:43443): tablet f6e363df7380414987ce798c1726adcd (table test-table [id=2cd107b6c4d64e41b7b45a5acd8e2b0a]) successfully deleted
I20260929 10:54:43.244805 13488 tablet_replica.cc:330] T f6e363df7380414987ce798c1726adcd P aeb660dae67845ffafb7f377e67753f3: stopping tablet replica
I20260929 10:54:43.245013 13486 tablet_replica.cc:330] T f6e363df7380414987ce798c1726adcd P f2a14a2d305643c195076b4d55901271: stopping tablet replica
I20260929 10:54:43.245149 13486 raft_consensus.cc:2229] T f6e363df7380414987ce798c1726adcd P f2a14a2d305643c195076b4d55901271 [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:54:43.245317 13486 raft_consensus.cc:2258] T f6e363df7380414987ce798c1726adcd P f2a14a2d305643c195076b4d55901271 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:54:43.245509 13488 raft_consensus.cc:2229] T f6e363df7380414987ce798c1726adcd P aeb660dae67845ffafb7f377e67753f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:54:43.245647 13488 raft_consensus.cc:2258] T f6e363df7380414987ce798c1726adcd P aeb660dae67845ffafb7f377e67753f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:54:43.245751 13486 ts_tablet_manager.cc:1905] T f6e363df7380414987ce798c1726adcd P f2a14a2d305643c195076b4d55901271: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:54:43.246172 13488 ts_tablet_manager.cc:1905] T f6e363df7380414987ce798c1726adcd P aeb660dae67845ffafb7f377e67753f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:54:43.247682 13486 ts_tablet_manager.cc:1918] T f6e363df7380414987ce798c1726adcd P f2a14a2d305643c195076b4d55901271: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 10:54:43.247829 13486 log.cc:1196] T f6e363df7380414987ce798c1726adcd P f2a14a2d305643c195076b4d55901271: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-1/wal/wals/f6e363df7380414987ce798c1726adcd
I20260929 10:54:43.248039 13488 ts_tablet_manager.cc:1918] T f6e363df7380414987ce798c1726adcd P aeb660dae67845ffafb7f377e67753f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 10:54:43.248131 13486 ts_tablet_manager.cc:1939] T f6e363df7380414987ce798c1726adcd P f2a14a2d305643c195076b4d55901271: Deleting consensus metadata
I20260929 10:54:43.248183 13488 log.cc:1196] T f6e363df7380414987ce798c1726adcd P aeb660dae67845ffafb7f377e67753f3: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-2/wal/wals/f6e363df7380414987ce798c1726adcd
I20260929 10:54:43.248565 12917 catalog_manager.cc:4841] TS f2a14a2d305643c195076b4d55901271 (127.10.238.194:38251): tablet f6e363df7380414987ce798c1726adcd (table test-table [id=2cd107b6c4d64e41b7b45a5acd8e2b0a]) successfully deleted
I20260929 10:54:43.248678 13488 ts_tablet_manager.cc:1939] T f6e363df7380414987ce798c1726adcd P aeb660dae67845ffafb7f377e67753f3: Deleting consensus metadata
I20260929 10:54:43.249128 12916 catalog_manager.cc:4841] TS aeb660dae67845ffafb7f377e67753f3 (127.10.238.195:36733): tablet f6e363df7380414987ce798c1726adcd (table test-table [id=2cd107b6c4d64e41b7b45a5acd8e2b0a]) successfully deleted
I20260929 10:54:43.531847 13445 raft_consensus.cc:482] T 722af37adf4c4a579563e562965f07b4 P f2a14a2d305643c195076b4d55901271 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:54:43.531952 13445 raft_consensus.cc:504] T 722af37adf4c4a579563e562965f07b4 P f2a14a2d305643c195076b4d55901271 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } }
I20260929 10:54:43.532111 13445 leader_election.cc:288] T 722af37adf4c4a579563e562965f07b4 P f2a14a2d305643c195076b4d55901271 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ba1f180c87d40bdb61d85c762d7e342 (127.10.238.193:43443), aeb660dae67845ffafb7f377e67753f3 (127.10.238.195:36733)
I20260929 10:54:43.532331 13088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "722af37adf4c4a579563e562965f07b4" candidate_uuid: "f2a14a2d305643c195076b4d55901271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ba1f180c87d40bdb61d85c762d7e342" is_pre_election: true
I20260929 10:54:43.532363 13368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "722af37adf4c4a579563e562965f07b4" candidate_uuid: "f2a14a2d305643c195076b4d55901271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeb660dae67845ffafb7f377e67753f3" is_pre_election: true
I20260929 10:54:43.532441 13368 raft_consensus.cc:2454] T 722af37adf4c4a579563e562965f07b4 P aeb660dae67845ffafb7f377e67753f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2a14a2d305643c195076b4d55901271 in term 0.
I20260929 10:54:43.532450 13088 raft_consensus.cc:2454] T 722af37adf4c4a579563e562965f07b4 P 9ba1f180c87d40bdb61d85c762d7e342 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2a14a2d305643c195076b4d55901271 in term 0.
I20260929 10:54:43.532611 13161 leader_election.cc:302] T 722af37adf4c4a579563e562965f07b4 P f2a14a2d305643c195076b4d55901271 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ba1f180c87d40bdb61d85c762d7e342, f2a14a2d305643c195076b4d55901271; no voters:
I20260929 10:54:43.532749 13445 raft_consensus.cc:2781] T 722af37adf4c4a579563e562965f07b4 P f2a14a2d305643c195076b4d55901271 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:54:43.532807 13445 raft_consensus.cc:482] T 722af37adf4c4a579563e562965f07b4 P f2a14a2d305643c195076b4d55901271 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:54:43.532823 13445 raft_consensus.cc:3037] T 722af37adf4c4a579563e562965f07b4 P f2a14a2d305643c195076b4d55901271 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:54:43.533538 13445 raft_consensus.cc:504] T 722af37adf4c4a579563e562965f07b4 P f2a14a2d305643c195076b4d55901271 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } }
I20260929 10:54:43.533680 13445 leader_election.cc:288] T 722af37adf4c4a579563e562965f07b4 P f2a14a2d305643c195076b4d55901271 [CANDIDATE]: Term 1 election: Requested vote from peers 9ba1f180c87d40bdb61d85c762d7e342 (127.10.238.193:43443), aeb660dae67845ffafb7f377e67753f3 (127.10.238.195:36733)
I20260929 10:54:43.533849 13088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "722af37adf4c4a579563e562965f07b4" candidate_uuid: "f2a14a2d305643c195076b4d55901271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ba1f180c87d40bdb61d85c762d7e342"
I20260929 10:54:43.533851 13368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "722af37adf4c4a579563e562965f07b4" candidate_uuid: "f2a14a2d305643c195076b4d55901271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeb660dae67845ffafb7f377e67753f3"
I20260929 10:54:43.533926 13088 raft_consensus.cc:3037] T 722af37adf4c4a579563e562965f07b4 P 9ba1f180c87d40bdb61d85c762d7e342 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:54:43.533926 13368 raft_consensus.cc:3037] T 722af37adf4c4a579563e562965f07b4 P aeb660dae67845ffafb7f377e67753f3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:54:43.534669 13368 raft_consensus.cc:2454] T 722af37adf4c4a579563e562965f07b4 P aeb660dae67845ffafb7f377e67753f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2a14a2d305643c195076b4d55901271 in term 1.
I20260929 10:54:43.534749 13088 raft_consensus.cc:2454] T 722af37adf4c4a579563e562965f07b4 P 9ba1f180c87d40bdb61d85c762d7e342 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2a14a2d305643c195076b4d55901271 in term 1.
I20260929 10:54:43.534878 13160 leader_election.cc:302] T 722af37adf4c4a579563e562965f07b4 P f2a14a2d305643c195076b4d55901271 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aeb660dae67845ffafb7f377e67753f3, f2a14a2d305643c195076b4d55901271; no voters:
I20260929 10:54:43.535008 13445 raft_consensus.cc:2781] T 722af37adf4c4a579563e562965f07b4 P f2a14a2d305643c195076b4d55901271 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:54:43.535094 13445 raft_consensus.cc:686] T 722af37adf4c4a579563e562965f07b4 P f2a14a2d305643c195076b4d55901271 [term 1 LEADER]: Becoming Leader. State: Replica: f2a14a2d305643c195076b4d55901271, State: Running, Role: LEADER
I20260929 10:54:43.535151 13445 consensus_queue.cc:237] T 722af37adf4c4a579563e562965f07b4 P f2a14a2d305643c195076b4d55901271 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } }
I20260929 10:54:43.535789 12931 catalog_manager.cc:5495] T 722af37adf4c4a579563e562965f07b4 P f2a14a2d305643c195076b4d55901271 reported cstate change: term changed from 0 to 1, leader changed from <none> to f2a14a2d305643c195076b4d55901271 (127.10.238.194). New cstate: current_term: 1 leader_uuid: "f2a14a2d305643c195076b4d55901271" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } health_report { overall_health: UNKNOWN } } }
I20260929 10:54:43.590225 13368 raft_consensus.cc:1261] T 722af37adf4c4a579563e562965f07b4 P aeb660dae67845ffafb7f377e67753f3 [term 1 FOLLOWER]: Refusing update from remote peer f2a14a2d305643c195076b4d55901271: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:54:43.590236 13088 raft_consensus.cc:1261] T 722af37adf4c4a579563e562965f07b4 P 9ba1f180c87d40bdb61d85c762d7e342 [term 1 FOLLOWER]: Refusing update from remote peer f2a14a2d305643c195076b4d55901271: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:54:43.590440 13445 consensus_queue.cc:1035] T 722af37adf4c4a579563e562965f07b4 P f2a14a2d305643c195076b4d55901271 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:54:43.590553 13447 consensus_queue.cc:1035] T 722af37adf4c4a579563e562965f07b4 P f2a14a2d305643c195076b4d55901271 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 10:54:43.594767 13467 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 10:54:45.112362 12929 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:54:45.112468 12929 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:54:45.113395 12929 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a789ec90b43c4dfe890712496bbac12f: Sending DeleteTablet for 3 replicas of tablet 722af37adf4c4a579563e562965f07b4
I20260929 10:54:45.113698 13348 tablet_service.cc:1514] Processing DeleteTablet for tablet 722af37adf4c4a579563e562965f07b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:54:45 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:33068
I20260929 10:54:45.113929 13517 tablet_replica.cc:330] T 722af37adf4c4a579563e562965f07b4 P aeb660dae67845ffafb7f377e67753f3: stopping tablet replica
I20260929 10:54:45.114010 13517 raft_consensus.cc:2229] T 722af37adf4c4a579563e562965f07b4 P aeb660dae67845ffafb7f377e67753f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:54:45.114056 13517 raft_consensus.cc:2258] T 722af37adf4c4a579563e562965f07b4 P aeb660dae67845ffafb7f377e67753f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:54:45.114274 13517 ts_tablet_manager.cc:1905] T 722af37adf4c4a579563e562965f07b4 P aeb660dae67845ffafb7f377e67753f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:54:45.115376 13068 tablet_service.cc:1514] Processing DeleteTablet for tablet 722af37adf4c4a579563e562965f07b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:54:45 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:40136
I20260929 10:54:45.115584 13201 tablet_service.cc:1514] Processing DeleteTablet for tablet 722af37adf4c4a579563e562965f07b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:54:45 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:34696
I20260929 10:54:45.115881 13517 ts_tablet_manager.cc:1918] T 722af37adf4c4a579563e562965f07b4 P aeb660dae67845ffafb7f377e67753f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 10:54:45.115957 13517 log.cc:1196] T 722af37adf4c4a579563e562965f07b4 P aeb660dae67845ffafb7f377e67753f3: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-2/wal/wals/722af37adf4c4a579563e562965f07b4
I20260929 10:54:45.116199 13517 ts_tablet_manager.cc:1939] T 722af37adf4c4a579563e562965f07b4 P aeb660dae67845ffafb7f377e67753f3: Deleting consensus metadata
I20260929 10:54:45.116546 12916 catalog_manager.cc:4841] TS aeb660dae67845ffafb7f377e67753f3 (127.10.238.195:36733): tablet 722af37adf4c4a579563e562965f07b4 (table test-table [id=0536518b664e481e9c0367058b3da09e]) successfully deleted
I20260929 10:54:45.116912 13520 tablet_replica.cc:330] T 722af37adf4c4a579563e562965f07b4 P f2a14a2d305643c195076b4d55901271: stopping tablet replica
I20260929 10:54:45.116992 13520 raft_consensus.cc:2229] T 722af37adf4c4a579563e562965f07b4 P f2a14a2d305643c195076b4d55901271 [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:54:45.117063 13520 raft_consensus.cc:2258] T 722af37adf4c4a579563e562965f07b4 P f2a14a2d305643c195076b4d55901271 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:54:45.117167 13519 tablet_replica.cc:330] T 722af37adf4c4a579563e562965f07b4 P 9ba1f180c87d40bdb61d85c762d7e342: stopping tablet replica
I20260929 10:54:45.117249 13519 raft_consensus.cc:2229] T 722af37adf4c4a579563e562965f07b4 P 9ba1f180c87d40bdb61d85c762d7e342 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:54:45.117322 13519 raft_consensus.cc:2258] T 722af37adf4c4a579563e562965f07b4 P 9ba1f180c87d40bdb61d85c762d7e342 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:54:45.117555 13520 ts_tablet_manager.cc:1905] T 722af37adf4c4a579563e562965f07b4 P f2a14a2d305643c195076b4d55901271: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:54:45.118846 13520 ts_tablet_manager.cc:1918] T 722af37adf4c4a579563e562965f07b4 P f2a14a2d305643c195076b4d55901271: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 10:54:45.118911 13520 log.cc:1196] T 722af37adf4c4a579563e562965f07b4 P f2a14a2d305643c195076b4d55901271: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-1/wal/wals/722af37adf4c4a579563e562965f07b4
I20260929 10:54:45.119159 13520 ts_tablet_manager.cc:1939] T 722af37adf4c4a579563e562965f07b4 P f2a14a2d305643c195076b4d55901271: Deleting consensus metadata
I20260929 10:54:45.119488 12917 catalog_manager.cc:4841] TS f2a14a2d305643c195076b4d55901271 (127.10.238.194:38251): tablet 722af37adf4c4a579563e562965f07b4 (table test-table [id=0536518b664e481e9c0367058b3da09e]) successfully deleted
I20260929 10:54:45.119537 13519 ts_tablet_manager.cc:1905] T 722af37adf4c4a579563e562965f07b4 P 9ba1f180c87d40bdb61d85c762d7e342: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:54:45.120869 13519 ts_tablet_manager.cc:1918] T 722af37adf4c4a579563e562965f07b4 P 9ba1f180c87d40bdb61d85c762d7e342: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 10:54:45.120949 13519 log.cc:1196] T 722af37adf4c4a579563e562965f07b4 P 9ba1f180c87d40bdb61d85c762d7e342: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-0/wal/wals/722af37adf4c4a579563e562965f07b4
I20260929 10:54:45.121145 13519 ts_tablet_manager.cc:1939] T 722af37adf4c4a579563e562965f07b4 P 9ba1f180c87d40bdb61d85c762d7e342: Deleting consensus metadata
I20260929 10:54:45.121522 12917 catalog_manager.cc:4841] TS 9ba1f180c87d40bdb61d85c762d7e342 (127.10.238.193:43443): tablet 722af37adf4c4a579563e562965f07b4 (table test-table [id=0536518b664e481e9c0367058b3da09e]) successfully deleted
Sep 29 10:54:45 dist-test-slave-8nrt krb5kdc[12886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679285, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 10:54:45 dist-test-slave-8nrt krb5kdc[12886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679285, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:54:45.136978 13527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:45.130220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38674 (local address 127.10.238.254:41661)
0929 10:54:45.130362 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:45.130364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:45.130404 (+ 40us) server_negotiation.cc:407] Connection header received
0929 10:54:45.130461 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:45.130461 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:45.130472 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:45.130539 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:45.131102 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:45.131609 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:45.132315 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:45.132509 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:45.132555 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:45.134672 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:45.134674 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:45.134674 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:45.134680 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:45.135966 (+ 1286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:45.136312 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:45.136313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:45.136313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:45.136373 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:45.136535 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:45.136536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:45.136536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:45.136635 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:45.136710 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:45.136844 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 10:54:45.136911 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":84,"threads_started":1}
I20260929 10:54:45.138393 12929 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38674:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:54:45.138557 12929 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:54:45.139910 13068 tablet_service.cc:1467] Processing CreateTablet for tablet 755e43e5463c4b88acc686de090dc068 (DEFAULT_TABLE table=test-table [id=bf90f9fd646041038d86cc90dffd9b34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:54:45.139974 13348 tablet_service.cc:1467] Processing CreateTablet for tablet 755e43e5463c4b88acc686de090dc068 (DEFAULT_TABLE table=test-table [id=bf90f9fd646041038d86cc90dffd9b34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:54:45.139974 13201 tablet_service.cc:1467] Processing CreateTablet for tablet 755e43e5463c4b88acc686de090dc068 (DEFAULT_TABLE table=test-table [id=bf90f9fd646041038d86cc90dffd9b34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:54:45.140098 13348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 755e43e5463c4b88acc686de090dc068. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:54:45.140100 13201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 755e43e5463c4b88acc686de090dc068. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:54:45.140101 13068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 755e43e5463c4b88acc686de090dc068. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:54:45.141105 13528 tablet_bootstrap.cc:492] T 755e43e5463c4b88acc686de090dc068 P 9ba1f180c87d40bdb61d85c762d7e342: Bootstrap starting.
I20260929 10:54:45.141443 13529 tablet_bootstrap.cc:492] T 755e43e5463c4b88acc686de090dc068 P aeb660dae67845ffafb7f377e67753f3: Bootstrap starting.
I20260929 10:54:45.141592 13528 tablet_bootstrap.cc:654] T 755e43e5463c4b88acc686de090dc068 P 9ba1f180c87d40bdb61d85c762d7e342: Neither blocks nor log segments found. Creating new log.
I20260929 10:54:45.141709 13530 tablet_bootstrap.cc:492] T 755e43e5463c4b88acc686de090dc068 P f2a14a2d305643c195076b4d55901271: Bootstrap starting.
I20260929 10:54:45.141867 13529 tablet_bootstrap.cc:654] T 755e43e5463c4b88acc686de090dc068 P aeb660dae67845ffafb7f377e67753f3: Neither blocks nor log segments found. Creating new log.
I20260929 10:54:45.142127 13530 tablet_bootstrap.cc:654] T 755e43e5463c4b88acc686de090dc068 P f2a14a2d305643c195076b4d55901271: Neither blocks nor log segments found. Creating new log.
I20260929 10:54:45.142503 13528 tablet_bootstrap.cc:492] T 755e43e5463c4b88acc686de090dc068 P 9ba1f180c87d40bdb61d85c762d7e342: No bootstrap required, opened a new log
I20260929 10:54:45.142562 13528 ts_tablet_manager.cc:1397] T 755e43e5463c4b88acc686de090dc068 P 9ba1f180c87d40bdb61d85c762d7e342: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 10:54:45.142671 13528 raft_consensus.cc:348] T 755e43e5463c4b88acc686de090dc068 P 9ba1f180c87d40bdb61d85c762d7e342 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } }
I20260929 10:54:45.142716 13529 tablet_bootstrap.cc:492] T 755e43e5463c4b88acc686de090dc068 P aeb660dae67845ffafb7f377e67753f3: No bootstrap required, opened a new log
I20260929 10:54:45.142732 13528 raft_consensus.cc:374] T 755e43e5463c4b88acc686de090dc068 P 9ba1f180c87d40bdb61d85c762d7e342 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:54:45.142726 13530 tablet_bootstrap.cc:492] T 755e43e5463c4b88acc686de090dc068 P f2a14a2d305643c195076b4d55901271: No bootstrap required, opened a new log
I20260929 10:54:45.142746 13528 raft_consensus.cc:729] T 755e43e5463c4b88acc686de090dc068 P 9ba1f180c87d40bdb61d85c762d7e342 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ba1f180c87d40bdb61d85c762d7e342, State: Initialized, Role: FOLLOWER
I20260929 10:54:45.142761 13529 ts_tablet_manager.cc:1397] T 755e43e5463c4b88acc686de090dc068 P aeb660dae67845ffafb7f377e67753f3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 10:54:45.142761 13530 ts_tablet_manager.cc:1397] T 755e43e5463c4b88acc686de090dc068 P f2a14a2d305643c195076b4d55901271: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 10:54:45.142803 13528 consensus_queue.cc:260] T 755e43e5463c4b88acc686de090dc068 P 9ba1f180c87d40bdb61d85c762d7e342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } }
I20260929 10:54:45.142912 13528 ts_tablet_manager.cc:1428] T 755e43e5463c4b88acc686de090dc068 P 9ba1f180c87d40bdb61d85c762d7e342: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:45.142877 13530 raft_consensus.cc:348] T 755e43e5463c4b88acc686de090dc068 P f2a14a2d305643c195076b4d55901271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } }
I20260929 10:54:45.142961 13530 raft_consensus.cc:374] T 755e43e5463c4b88acc686de090dc068 P f2a14a2d305643c195076b4d55901271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:54:45.142983 13530 raft_consensus.cc:729] T 755e43e5463c4b88acc686de090dc068 P f2a14a2d305643c195076b4d55901271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2a14a2d305643c195076b4d55901271, State: Initialized, Role: FOLLOWER
I20260929 10:54:45.142916 13529 raft_consensus.cc:348] T 755e43e5463c4b88acc686de090dc068 P aeb660dae67845ffafb7f377e67753f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } }
I20260929 10:54:45.143005 13529 raft_consensus.cc:374] T 755e43e5463c4b88acc686de090dc068 P aeb660dae67845ffafb7f377e67753f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:54:45.143028 13529 raft_consensus.cc:729] T 755e43e5463c4b88acc686de090dc068 P aeb660dae67845ffafb7f377e67753f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeb660dae67845ffafb7f377e67753f3, State: Initialized, Role: FOLLOWER
I20260929 10:54:45.143033 13530 consensus_queue.cc:260] T 755e43e5463c4b88acc686de090dc068 P f2a14a2d305643c195076b4d55901271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } }
I20260929 10:54:45.143076 13529 consensus_queue.cc:260] T 755e43e5463c4b88acc686de090dc068 P aeb660dae67845ffafb7f377e67753f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } }
I20260929 10:54:45.143132 13530 ts_tablet_manager.cc:1428] T 755e43e5463c4b88acc686de090dc068 P f2a14a2d305643c195076b4d55901271: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:45.143225 13529 ts_tablet_manager.cc:1428] T 755e43e5463c4b88acc686de090dc068 P aeb660dae67845ffafb7f377e67753f3: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 10:54:45.208616 13534 raft_consensus.cc:482] T 755e43e5463c4b88acc686de090dc068 P aeb660dae67845ffafb7f377e67753f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:54:45.208704 13534 raft_consensus.cc:504] T 755e43e5463c4b88acc686de090dc068 P aeb660dae67845ffafb7f377e67753f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } }
I20260929 10:54:45.208981 13534 leader_election.cc:288] T 755e43e5463c4b88acc686de090dc068 P aeb660dae67845ffafb7f377e67753f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ba1f180c87d40bdb61d85c762d7e342 (127.10.238.193:43443), f2a14a2d305643c195076b4d55901271 (127.10.238.194:38251)
I20260929 10:54:45.213280 13538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:45.209507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:53101 (local address 127.10.238.194:38251)
0929 10:54:45.209662 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:45.209665 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:45.209672 (+ 7us) server_negotiation.cc:407] Connection header received
0929 10:54:45.209698 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:45.209699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:45.209729 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:45.209783 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:54:45.209953 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:45.210703 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:45.212258 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:45.212876 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:45.212948 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:45.212954 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:45.213151 (+ 197us) server_negotiation.cc:299] Negotiation successful
0929 10:54:45.213195 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":80,"threads_started":1}
I20260929 10:54:45.213292 13536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:45.209250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:52089 (local address 127.10.238.193:43443)
0929 10:54:45.209495 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:45.209497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:45.209506 (+ 9us) server_negotiation.cc:407] Connection header received
0929 10:54:45.209536 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:45.209537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:45.209547 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:45.209643 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:54:45.209855 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:45.210703 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:45.212258 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:45.212820 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:45.212891 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:45.212901 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:45.213149 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 10:54:45.213197 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":130,"threads_started":1}
I20260929 10:54:45.213304 13535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:45.209185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:43443 (local address 127.10.238.195:52089)
0929 10:54:45.209404 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:45.209416 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:45.209479 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:45.209674 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:45.209675 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:45.209677 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:54:45.209750 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:45.209752 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:45.210794 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:45.210795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:45.212164 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:45.212167 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:45.212939 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:45.212941 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:45.213031 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:45.213034 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 10:54:45.213111 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 10:54:45.213164 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":57,"thread_start_us":141,"threads_started":1}
I20260929 10:54:45.213310 13537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:45.209441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.194:38251 (local address 127.10.238.195:53101)
0929 10:54:45.209568 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:45.209577 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:45.209631 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:45.209805 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:45.209806 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:45.209808 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:54:45.209884 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:45.209885 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:45.210788 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:45.210789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:45.212164 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:45.212167 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:45.212949 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:45.212950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:45.213025 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:45.213029 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 10:54:45.213125 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 10:54:45.213173 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":93,"thread_start_us":64,"threads_started":1}
I20260929 10:54:45.213543 13088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "755e43e5463c4b88acc686de090dc068" candidate_uuid: "aeb660dae67845ffafb7f377e67753f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ba1f180c87d40bdb61d85c762d7e342" is_pre_election: true
I20260929 10:54:45.213627 13088 raft_consensus.cc:2454] T 755e43e5463c4b88acc686de090dc068 P 9ba1f180c87d40bdb61d85c762d7e342 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aeb660dae67845ffafb7f377e67753f3 in term 0.
I20260929 10:54:45.213800 13301 leader_election.cc:302] T 755e43e5463c4b88acc686de090dc068 P aeb660dae67845ffafb7f377e67753f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ba1f180c87d40bdb61d85c762d7e342, aeb660dae67845ffafb7f377e67753f3; no voters:
I20260929 10:54:45.213825 13228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "755e43e5463c4b88acc686de090dc068" candidate_uuid: "aeb660dae67845ffafb7f377e67753f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2a14a2d305643c195076b4d55901271" is_pre_election: true
I20260929 10:54:45.213914 13228 raft_consensus.cc:2454] T 755e43e5463c4b88acc686de090dc068 P f2a14a2d305643c195076b4d55901271 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aeb660dae67845ffafb7f377e67753f3 in term 0.
I20260929 10:54:45.213979 13534 raft_consensus.cc:2781] T 755e43e5463c4b88acc686de090dc068 P aeb660dae67845ffafb7f377e67753f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:54:45.214032 13534 raft_consensus.cc:482] T 755e43e5463c4b88acc686de090dc068 P aeb660dae67845ffafb7f377e67753f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:54:45.214071 13534 raft_consensus.cc:3037] T 755e43e5463c4b88acc686de090dc068 P aeb660dae67845ffafb7f377e67753f3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:54:45.214784 13534 raft_consensus.cc:504] T 755e43e5463c4b88acc686de090dc068 P aeb660dae67845ffafb7f377e67753f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } }
I20260929 10:54:45.214915 13534 leader_election.cc:288] T 755e43e5463c4b88acc686de090dc068 P aeb660dae67845ffafb7f377e67753f3 [CANDIDATE]: Term 1 election: Requested vote from peers 9ba1f180c87d40bdb61d85c762d7e342 (127.10.238.193:43443), f2a14a2d305643c195076b4d55901271 (127.10.238.194:38251)
I20260929 10:54:45.215085 13088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "755e43e5463c4b88acc686de090dc068" candidate_uuid: "aeb660dae67845ffafb7f377e67753f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ba1f180c87d40bdb61d85c762d7e342"
I20260929 10:54:45.215116 13228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "755e43e5463c4b88acc686de090dc068" candidate_uuid: "aeb660dae67845ffafb7f377e67753f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2a14a2d305643c195076b4d55901271"
I20260929 10:54:45.215183 13088 raft_consensus.cc:3037] T 755e43e5463c4b88acc686de090dc068 P 9ba1f180c87d40bdb61d85c762d7e342 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:54:45.215186 13228 raft_consensus.cc:3037] T 755e43e5463c4b88acc686de090dc068 P f2a14a2d305643c195076b4d55901271 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:54:45.215775 13088 raft_consensus.cc:2454] T 755e43e5463c4b88acc686de090dc068 P 9ba1f180c87d40bdb61d85c762d7e342 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aeb660dae67845ffafb7f377e67753f3 in term 1.
I20260929 10:54:45.215909 13228 raft_consensus.cc:2454] T 755e43e5463c4b88acc686de090dc068 P f2a14a2d305643c195076b4d55901271 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aeb660dae67845ffafb7f377e67753f3 in term 1.
I20260929 10:54:45.216007 13301 leader_election.cc:302] T 755e43e5463c4b88acc686de090dc068 P aeb660dae67845ffafb7f377e67753f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ba1f180c87d40bdb61d85c762d7e342, aeb660dae67845ffafb7f377e67753f3; no voters:
I20260929 10:54:45.216106 13534 raft_consensus.cc:2781] T 755e43e5463c4b88acc686de090dc068 P aeb660dae67845ffafb7f377e67753f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:54:45.216269 13534 raft_consensus.cc:686] T 755e43e5463c4b88acc686de090dc068 P aeb660dae67845ffafb7f377e67753f3 [term 1 LEADER]: Becoming Leader. State: Replica: aeb660dae67845ffafb7f377e67753f3, State: Running, Role: LEADER
I20260929 10:54:45.216325 13534 consensus_queue.cc:237] T 755e43e5463c4b88acc686de090dc068 P aeb660dae67845ffafb7f377e67753f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } }
I20260929 10:54:45.217073 12930 catalog_manager.cc:5495] T 755e43e5463c4b88acc686de090dc068 P aeb660dae67845ffafb7f377e67753f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to aeb660dae67845ffafb7f377e67753f3 (127.10.238.195). New cstate: current_term: 1 leader_uuid: "aeb660dae67845ffafb7f377e67753f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } health_report { overall_health: HEALTHY } } }
I20260929 10:54:45.237179 13542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:45.234840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33072 (local address 127.10.238.195:36733)
0929 10:54:45.234999 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:45.235001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:45.235008 (+ 7us) server_negotiation.cc:407] Connection header received
0929 10:54:45.235040 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:45.235041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:45.235058 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:45.235125 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:54:45.235248 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:45.235743 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:45.236479 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:45.236667 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:45.236715 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:45.236854 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:54:45.236954 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:54:45.237004 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:45.237043 (+ 39us) server_negotiation.cc:299] Negotiation successful
0929 10:54:45.237058 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":93,"threads_started":1}
I20260929 10:54:45.241466 13536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:45.239406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:48987 (local address 127.10.238.193:43443)
0929 10:54:45.239443 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:45.239445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:45.239542 (+ 97us) server_negotiation.cc:407] Connection header received
0929 10:54:45.239567 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:45.239567 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:45.239574 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:45.239618 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:54:45.239708 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:45.240131 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:45.240852 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:45.241021 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:45.241067 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:45.241209 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:54:45.241233 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:54:45.241343 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:45.241400 (+ 57us) server_negotiation.cc:299] Negotiation successful
0929 10:54:45.241416 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 10:54:45.241490 13545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:45.239323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:43443 (local address 127.10.238.195:48987)
0929 10:54:45.239470 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:45.239479 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:45.239536 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:45.239620 (+ 84us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:45.239620 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:45.239622 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 10:54:45.239672 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:45.239673 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:45.240205 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:45.240205 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:45.240775 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:45.240777 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:45.241087 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:45.241087 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:45.241131 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:45.241143 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 10:54:45.241355 (+ 212us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 10:54:45.241361 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 10:54:45.241396 (+ 35us) client_negotiation.cc:240] Negotiation successful
0929 10:54:45.241430 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":85,"threads_started":1}
I20260929 10:54:45.243492 13088 raft_consensus.cc:1261] T 755e43e5463c4b88acc686de090dc068 P 9ba1f180c87d40bdb61d85c762d7e342 [term 1 FOLLOWER]: Refusing update from remote peer aeb660dae67845ffafb7f377e67753f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:54:45.243480 13228 raft_consensus.cc:1261] T 755e43e5463c4b88acc686de090dc068 P f2a14a2d305643c195076b4d55901271 [term 1 FOLLOWER]: Refusing update from remote peer aeb660dae67845ffafb7f377e67753f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:54:45.243755 13539 consensus_queue.cc:1035] T 755e43e5463c4b88acc686de090dc068 P aeb660dae67845ffafb7f377e67753f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:54:45.243870 13534 consensus_queue.cc:1035] T 755e43e5463c4b88acc686de090dc068 P aeb660dae67845ffafb7f377e67753f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:54:45.250976 13542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:45.248819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:60171 (local address 127.10.238.195:36733)
0929 10:54:45.248856 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:45.248857 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:45.248931 (+ 74us) server_negotiation.cc:407] Connection header received
0929 10:54:45.248967 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:45.248968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:45.248975 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:45.249019 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:54:45.249092 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:45.249601 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:45.250260 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:45.250452 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:45.250516 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:45.250760 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:54:45.250794 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:54:45.250837 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:45.250899 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 10:54:45.250919 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 10:54:45.251025 13550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:45.248724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.195:36733 (local address 127.10.238.193:60171)
0929 10:54:45.248851 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:45.248862 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:45.248918 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:45.249020 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:45.249020 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:45.249021 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 10:54:45.249058 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:45.249059 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:45.249661 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:45.249661 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:45.250175 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:45.250176 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:45.250539 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:45.250540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:45.250678 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:45.250685 (+ 7us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 10:54:45.250855 (+ 170us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 10:54:45.250857 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 10:54:45.250898 (+ 41us) client_negotiation.cc:240] Negotiation successful
0929 10:54:45.250943 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":80,"threads_started":1}
I20260929 10:54:45.257325 12930 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:54:45.257437 12930 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:54:45.258097 12930 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a789ec90b43c4dfe890712496bbac12f: Sending DeleteTablet for 3 replicas of tablet 755e43e5463c4b88acc686de090dc068
I20260929 10:54:45.258353 13348 tablet_service.cc:1514] Processing DeleteTablet for tablet 755e43e5463c4b88acc686de090dc068 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:54:45 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:33068
I20260929 10:54:45.258396 13201 tablet_service.cc:1514] Processing DeleteTablet for tablet 755e43e5463c4b88acc686de090dc068 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:54:45 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:34696
I20260929 10:54:45.258395 13068 tablet_service.cc:1514] Processing DeleteTablet for tablet 755e43e5463c4b88acc686de090dc068 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:54:45 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:40136
I20260929 10:54:45.258430 13517 tablet_replica.cc:330] T 755e43e5463c4b88acc686de090dc068 P aeb660dae67845ffafb7f377e67753f3: stopping tablet replica
I20260929 10:54:45.258471 13517 raft_consensus.cc:2229] T 755e43e5463c4b88acc686de090dc068 P aeb660dae67845ffafb7f377e67753f3 [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:54:45.258481 13520 tablet_replica.cc:330] T 755e43e5463c4b88acc686de090dc068 P f2a14a2d305643c195076b4d55901271: stopping tablet replica
I20260929 10:54:45.258481 13519 tablet_replica.cc:330] T 755e43e5463c4b88acc686de090dc068 P 9ba1f180c87d40bdb61d85c762d7e342: stopping tablet replica
I20260929 10:54:45.258524 13520 raft_consensus.cc:2229] T 755e43e5463c4b88acc686de090dc068 P f2a14a2d305643c195076b4d55901271 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:54:45.258530 13519 raft_consensus.cc:2229] T 755e43e5463c4b88acc686de090dc068 P 9ba1f180c87d40bdb61d85c762d7e342 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:54:45.258540 13517 raft_consensus.cc:2258] T 755e43e5463c4b88acc686de090dc068 P aeb660dae67845ffafb7f377e67753f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:54:45.258559 13520 raft_consensus.cc:2258] T 755e43e5463c4b88acc686de090dc068 P f2a14a2d305643c195076b4d55901271 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:54:45.258569 13519 raft_consensus.cc:2258] T 755e43e5463c4b88acc686de090dc068 P 9ba1f180c87d40bdb61d85c762d7e342 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:54:45.258917 13519 ts_tablet_manager.cc:1905] T 755e43e5463c4b88acc686de090dc068 P 9ba1f180c87d40bdb61d85c762d7e342: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:54:45.258972 13517 ts_tablet_manager.cc:1905] T 755e43e5463c4b88acc686de090dc068 P aeb660dae67845ffafb7f377e67753f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:54:45.259078 12930 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38674:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:54:45.259213 12930 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:54:45.260689 13520 ts_tablet_manager.cc:1905] T 755e43e5463c4b88acc686de090dc068 P f2a14a2d305643c195076b4d55901271: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:54:45.260710 13517 ts_tablet_manager.cc:1918] T 755e43e5463c4b88acc686de090dc068 P aeb660dae67845ffafb7f377e67753f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 10:54:45.260757 13517 log.cc:1196] T 755e43e5463c4b88acc686de090dc068 P aeb660dae67845ffafb7f377e67753f3: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-2/wal/wals/755e43e5463c4b88acc686de090dc068
I20260929 10:54:45.260974 13517 ts_tablet_manager.cc:1939] T 755e43e5463c4b88acc686de090dc068 P aeb660dae67845ffafb7f377e67753f3: Deleting consensus metadata
I20260929 10:54:45.261309 12916 catalog_manager.cc:4841] TS aeb660dae67845ffafb7f377e67753f3 (127.10.238.195:36733): tablet 755e43e5463c4b88acc686de090dc068 (table test-table [id=bf90f9fd646041038d86cc90dffd9b34]) successfully deleted
I20260929 10:54:45.261417 13348 tablet_service.cc:1467] Processing CreateTablet for tablet 089e7119ddef4191bcc2c069e80ff066 (DEFAULT_TABLE table=test-table [id=7f2e8f06cd934f6b892d198d032755ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:54:45.261512 13348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 089e7119ddef4191bcc2c069e80ff066. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:54:45.261538 13519 ts_tablet_manager.cc:1918] T 755e43e5463c4b88acc686de090dc068 P 9ba1f180c87d40bdb61d85c762d7e342: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 10:54:45.261591 13519 log.cc:1196] T 755e43e5463c4b88acc686de090dc068 P 9ba1f180c87d40bdb61d85c762d7e342: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-0/wal/wals/755e43e5463c4b88acc686de090dc068
I20260929 10:54:45.261812 13519 ts_tablet_manager.cc:1939] T 755e43e5463c4b88acc686de090dc068 P 9ba1f180c87d40bdb61d85c762d7e342: Deleting consensus metadata
I20260929 10:54:45.262094 12917 catalog_manager.cc:4841] TS 9ba1f180c87d40bdb61d85c762d7e342 (127.10.238.193:43443): tablet 755e43e5463c4b88acc686de090dc068 (table test-table [id=bf90f9fd646041038d86cc90dffd9b34]) successfully deleted
I20260929 10:54:45.262355 13520 ts_tablet_manager.cc:1918] T 755e43e5463c4b88acc686de090dc068 P f2a14a2d305643c195076b4d55901271: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 10:54:45.262403 13520 log.cc:1196] T 755e43e5463c4b88acc686de090dc068 P f2a14a2d305643c195076b4d55901271: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-1/wal/wals/755e43e5463c4b88acc686de090dc068
I20260929 10:54:45.262601 13520 ts_tablet_manager.cc:1939] T 755e43e5463c4b88acc686de090dc068 P f2a14a2d305643c195076b4d55901271: Deleting consensus metadata
I20260929 10:54:45.262763 13529 tablet_bootstrap.cc:492] T 089e7119ddef4191bcc2c069e80ff066 P aeb660dae67845ffafb7f377e67753f3: Bootstrap starting.
I20260929 10:54:45.262869 13068 tablet_service.cc:1467] Processing CreateTablet for tablet 089e7119ddef4191bcc2c069e80ff066 (DEFAULT_TABLE table=test-table [id=7f2e8f06cd934f6b892d198d032755ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:54:45.262970 13068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 089e7119ddef4191bcc2c069e80ff066. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:54:45.263141 13529 tablet_bootstrap.cc:654] T 089e7119ddef4191bcc2c069e80ff066 P aeb660dae67845ffafb7f377e67753f3: Neither blocks nor log segments found. Creating new log.
I20260929 10:54:45.264024 13528 tablet_bootstrap.cc:492] T 089e7119ddef4191bcc2c069e80ff066 P 9ba1f180c87d40bdb61d85c762d7e342: Bootstrap starting.
I20260929 10:54:45.264063 13201 tablet_service.cc:1467] Processing CreateTablet for tablet 089e7119ddef4191bcc2c069e80ff066 (DEFAULT_TABLE table=test-table [id=7f2e8f06cd934f6b892d198d032755ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:54:45.264161 13201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 089e7119ddef4191bcc2c069e80ff066. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:54:45.264372 13528 tablet_bootstrap.cc:654] T 089e7119ddef4191bcc2c069e80ff066 P 9ba1f180c87d40bdb61d85c762d7e342: Neither blocks nor log segments found. Creating new log.
I20260929 10:54:45.264921 13528 tablet_bootstrap.cc:492] T 089e7119ddef4191bcc2c069e80ff066 P 9ba1f180c87d40bdb61d85c762d7e342: No bootstrap required, opened a new log
I20260929 10:54:45.264963 13528 ts_tablet_manager.cc:1397] T 089e7119ddef4191bcc2c069e80ff066 P 9ba1f180c87d40bdb61d85c762d7e342: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 10:54:45.265070 13530 tablet_bootstrap.cc:492] T 089e7119ddef4191bcc2c069e80ff066 P f2a14a2d305643c195076b4d55901271: Bootstrap starting.
I20260929 10:54:45.265064 13528 raft_consensus.cc:348] T 089e7119ddef4191bcc2c069e80ff066 P 9ba1f180c87d40bdb61d85c762d7e342 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } }
I20260929 10:54:45.265122 13528 raft_consensus.cc:374] T 089e7119ddef4191bcc2c069e80ff066 P 9ba1f180c87d40bdb61d85c762d7e342 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:54:45.265143 13528 raft_consensus.cc:729] T 089e7119ddef4191bcc2c069e80ff066 P 9ba1f180c87d40bdb61d85c762d7e342 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ba1f180c87d40bdb61d85c762d7e342, State: Initialized, Role: FOLLOWER
I20260929 10:54:45.265161 12917 catalog_manager.cc:4841] TS f2a14a2d305643c195076b4d55901271 (127.10.238.194:38251): tablet 755e43e5463c4b88acc686de090dc068 (table test-table [id=bf90f9fd646041038d86cc90dffd9b34]) successfully deleted
I20260929 10:54:45.265187 13528 consensus_queue.cc:260] T 089e7119ddef4191bcc2c069e80ff066 P 9ba1f180c87d40bdb61d85c762d7e342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } }
I20260929 10:54:45.265312 13528 ts_tablet_manager.cc:1428] T 089e7119ddef4191bcc2c069e80ff066 P 9ba1f180c87d40bdb61d85c762d7e342: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 10:54:45.265434 13530 tablet_bootstrap.cc:654] T 089e7119ddef4191bcc2c069e80ff066 P f2a14a2d305643c195076b4d55901271: Neither blocks nor log segments found. Creating new log.
I20260929 10:54:45.266244 13529 tablet_bootstrap.cc:492] T 089e7119ddef4191bcc2c069e80ff066 P aeb660dae67845ffafb7f377e67753f3: No bootstrap required, opened a new log
I20260929 10:54:45.266307 13529 ts_tablet_manager.cc:1397] T 089e7119ddef4191bcc2c069e80ff066 P aeb660dae67845ffafb7f377e67753f3: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260929 10:54:45.266438 13530 tablet_bootstrap.cc:492] T 089e7119ddef4191bcc2c069e80ff066 P f2a14a2d305643c195076b4d55901271: No bootstrap required, opened a new log
I20260929 10:54:45.266489 13530 ts_tablet_manager.cc:1397] T 089e7119ddef4191bcc2c069e80ff066 P f2a14a2d305643c195076b4d55901271: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 10:54:45.266499 13529 raft_consensus.cc:348] T 089e7119ddef4191bcc2c069e80ff066 P aeb660dae67845ffafb7f377e67753f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } }
I20260929 10:54:45.266549 13529 raft_consensus.cc:374] T 089e7119ddef4191bcc2c069e80ff066 P aeb660dae67845ffafb7f377e67753f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:54:45.266562 13529 raft_consensus.cc:729] T 089e7119ddef4191bcc2c069e80ff066 P aeb660dae67845ffafb7f377e67753f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeb660dae67845ffafb7f377e67753f3, State: Initialized, Role: FOLLOWER
I20260929 10:54:45.266587 13530 raft_consensus.cc:348] T 089e7119ddef4191bcc2c069e80ff066 P f2a14a2d305643c195076b4d55901271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } }
I20260929 10:54:45.266647 13530 raft_consensus.cc:374] T 089e7119ddef4191bcc2c069e80ff066 P f2a14a2d305643c195076b4d55901271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:54:45.266620 13529 consensus_queue.cc:260] T 089e7119ddef4191bcc2c069e80ff066 P aeb660dae67845ffafb7f377e67753f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } }
I20260929 10:54:45.266695 13530 raft_consensus.cc:729] T 089e7119ddef4191bcc2c069e80ff066 P f2a14a2d305643c195076b4d55901271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2a14a2d305643c195076b4d55901271, State: Initialized, Role: FOLLOWER
I20260929 10:54:45.266752 13530 consensus_queue.cc:260] T 089e7119ddef4191bcc2c069e80ff066 P f2a14a2d305643c195076b4d55901271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } }
I20260929 10:54:45.266772 13529 ts_tablet_manager.cc:1428] T 089e7119ddef4191bcc2c069e80ff066 P aeb660dae67845ffafb7f377e67753f3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:45.266846 13530 ts_tablet_manager.cc:1428] T 089e7119ddef4191bcc2c069e80ff066 P f2a14a2d305643c195076b4d55901271: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 10:54:45.546195 13495 raft_consensus.cc:482] T 089e7119ddef4191bcc2c069e80ff066 P 9ba1f180c87d40bdb61d85c762d7e342 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:54:45.546315 13495 raft_consensus.cc:504] T 089e7119ddef4191bcc2c069e80ff066 P 9ba1f180c87d40bdb61d85c762d7e342 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } }
I20260929 10:54:45.546535 13495 leader_election.cc:288] T 089e7119ddef4191bcc2c069e80ff066 P 9ba1f180c87d40bdb61d85c762d7e342 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2a14a2d305643c195076b4d55901271 (127.10.238.194:38251), aeb660dae67845ffafb7f377e67753f3 (127.10.238.195:36733)
I20260929 10:54:45.546784 13368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "089e7119ddef4191bcc2c069e80ff066" candidate_uuid: "9ba1f180c87d40bdb61d85c762d7e342" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeb660dae67845ffafb7f377e67753f3" is_pre_election: true
I20260929 10:54:45.546803 13228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "089e7119ddef4191bcc2c069e80ff066" candidate_uuid: "9ba1f180c87d40bdb61d85c762d7e342" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2a14a2d305643c195076b4d55901271" is_pre_election: true
I20260929 10:54:45.546890 13368 raft_consensus.cc:2454] T 089e7119ddef4191bcc2c069e80ff066 P aeb660dae67845ffafb7f377e67753f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ba1f180c87d40bdb61d85c762d7e342 in term 0.
I20260929 10:54:45.546893 13228 raft_consensus.cc:2454] T 089e7119ddef4191bcc2c069e80ff066 P f2a14a2d305643c195076b4d55901271 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ba1f180c87d40bdb61d85c762d7e342 in term 0.
I20260929 10:54:45.547062 13021 leader_election.cc:302] T 089e7119ddef4191bcc2c069e80ff066 P 9ba1f180c87d40bdb61d85c762d7e342 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ba1f180c87d40bdb61d85c762d7e342, f2a14a2d305643c195076b4d55901271; no voters:
I20260929 10:54:45.547175 13495 raft_consensus.cc:2781] T 089e7119ddef4191bcc2c069e80ff066 P 9ba1f180c87d40bdb61d85c762d7e342 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:54:45.547240 13495 raft_consensus.cc:482] T 089e7119ddef4191bcc2c069e80ff066 P 9ba1f180c87d40bdb61d85c762d7e342 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:54:45.547282 13495 raft_consensus.cc:3037] T 089e7119ddef4191bcc2c069e80ff066 P 9ba1f180c87d40bdb61d85c762d7e342 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:54:45.548219 13495 raft_consensus.cc:504] T 089e7119ddef4191bcc2c069e80ff066 P 9ba1f180c87d40bdb61d85c762d7e342 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } }
I20260929 10:54:45.548377 13495 leader_election.cc:288] T 089e7119ddef4191bcc2c069e80ff066 P 9ba1f180c87d40bdb61d85c762d7e342 [CANDIDATE]: Term 1 election: Requested vote from peers f2a14a2d305643c195076b4d55901271 (127.10.238.194:38251), aeb660dae67845ffafb7f377e67753f3 (127.10.238.195:36733)
I20260929 10:54:45.548520 13228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "089e7119ddef4191bcc2c069e80ff066" candidate_uuid: "9ba1f180c87d40bdb61d85c762d7e342" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2a14a2d305643c195076b4d55901271"
I20260929 10:54:45.548547 13368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "089e7119ddef4191bcc2c069e80ff066" candidate_uuid: "9ba1f180c87d40bdb61d85c762d7e342" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeb660dae67845ffafb7f377e67753f3"
I20260929 10:54:45.548614 13368 raft_consensus.cc:3037] T 089e7119ddef4191bcc2c069e80ff066 P aeb660dae67845ffafb7f377e67753f3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:54:45.548591 13228 raft_consensus.cc:3037] T 089e7119ddef4191bcc2c069e80ff066 P f2a14a2d305643c195076b4d55901271 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:54:45.549243 13368 raft_consensus.cc:2454] T 089e7119ddef4191bcc2c069e80ff066 P aeb660dae67845ffafb7f377e67753f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ba1f180c87d40bdb61d85c762d7e342 in term 1.
I20260929 10:54:45.549335 13228 raft_consensus.cc:2454] T 089e7119ddef4191bcc2c069e80ff066 P f2a14a2d305643c195076b4d55901271 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ba1f180c87d40bdb61d85c762d7e342 in term 1.
I20260929 10:54:45.549469 13020 leader_election.cc:302] T 089e7119ddef4191bcc2c069e80ff066 P 9ba1f180c87d40bdb61d85c762d7e342 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ba1f180c87d40bdb61d85c762d7e342, aeb660dae67845ffafb7f377e67753f3; no voters:
I20260929 10:54:45.549589 13495 raft_consensus.cc:2781] T 089e7119ddef4191bcc2c069e80ff066 P 9ba1f180c87d40bdb61d85c762d7e342 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:54:45.549669 13495 raft_consensus.cc:686] T 089e7119ddef4191bcc2c069e80ff066 P 9ba1f180c87d40bdb61d85c762d7e342 [term 1 LEADER]: Becoming Leader. State: Replica: 9ba1f180c87d40bdb61d85c762d7e342, State: Running, Role: LEADER
I20260929 10:54:45.549702 13495 consensus_queue.cc:237] T 089e7119ddef4191bcc2c069e80ff066 P 9ba1f180c87d40bdb61d85c762d7e342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } } peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } }
I20260929 10:54:45.550282 12929 catalog_manager.cc:5495] T 089e7119ddef4191bcc2c069e80ff066 P 9ba1f180c87d40bdb61d85c762d7e342 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ba1f180c87d40bdb61d85c762d7e342 (127.10.238.193). New cstate: current_term: 1 leader_uuid: "9ba1f180c87d40bdb61d85c762d7e342" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba1f180c87d40bdb61d85c762d7e342" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43443 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 } health_report { overall_health: UNKNOWN } } }
I20260929 10:54:45.624544 13536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:45.621895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54954 (local address 127.10.238.193:43443)
0929 10:54:45.621938 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:45.621940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:45.621967 (+ 27us) server_negotiation.cc:407] Connection header received
0929 10:54:45.622064 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:45.622064 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:45.622077 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:45.622133 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:54:45.622269 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:45.622735 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:45.623871 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:45.624069 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:45.624121 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:45.624298 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:54:45.624324 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:54:45.624362 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:45.624451 (+ 89us) server_negotiation.cc:299] Negotiation successful
0929 10:54:45.624469 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 10:54:45.628932 13536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:45.626838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:50279 (local address 127.10.238.193:43443)
0929 10:54:45.626867 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:45.626868 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:45.626874 (+ 6us) server_negotiation.cc:407] Connection header received
0929 10:54:45.626951 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:45.626952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:45.626959 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:45.627002 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:54:45.627111 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:45.627517 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:45.628288 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:45.628451 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:45.628497 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:45.628701 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:54:45.628731 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:54:45.628794 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:45.628855 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 10:54:45.628872 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":14}
I20260929 10:54:45.629031 13550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:45.626746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:43443 (local address 127.10.238.193:50279)
0929 10:54:45.626811 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:45.626829 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:45.626878 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:45.627020 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:45.627021 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:45.627022 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 10:54:45.627057 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:45.627058 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:45.627576 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:45.627576 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:45.628188 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:45.628189 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:45.628519 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:45.628520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:45.628565 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:45.628571 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 10:54:45.628800 (+ 229us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 10:54:45.628802 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 10:54:45.628853 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 10:54:45.628897 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":1}
I20260929 10:54:45.630915 13228 raft_consensus.cc:1261] T 089e7119ddef4191bcc2c069e80ff066 P f2a14a2d305643c195076b4d55901271 [term 1 FOLLOWER]: Refusing update from remote peer 9ba1f180c87d40bdb61d85c762d7e342: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:54:45.630915 13368 raft_consensus.cc:1261] T 089e7119ddef4191bcc2c069e80ff066 P aeb660dae67845ffafb7f377e67753f3 [term 1 FOLLOWER]: Refusing update from remote peer 9ba1f180c87d40bdb61d85c762d7e342: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:54:45.631153 13503 consensus_queue.cc:1035] T 089e7119ddef4191bcc2c069e80ff066 P 9ba1f180c87d40bdb61d85c762d7e342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb660dae67845ffafb7f377e67753f3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:54:45.631269 13498 consensus_queue.cc:1035] T 089e7119ddef4191bcc2c069e80ff066 P 9ba1f180c87d40bdb61d85c762d7e342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2a14a2d305643c195076b4d55901271" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 10:54:45.635537 13496 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 10:54:47.153833 12929 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:54:47.153934 12929 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:54:47.154734 12929 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a789ec90b43c4dfe890712496bbac12f: Sending DeleteTablet for 3 replicas of tablet 089e7119ddef4191bcc2c069e80ff066
I20260929 10:54:47.154994 13201 tablet_service.cc:1514] Processing DeleteTablet for tablet 089e7119ddef4191bcc2c069e80ff066 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:54:47 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:34696
I20260929 10:54:47.155006 13348 tablet_service.cc:1514] Processing DeleteTablet for tablet 089e7119ddef4191bcc2c069e80ff066 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:54:47 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:33068
I20260929 10:54:47.155196 13066 tablet_service.cc:1514] Processing DeleteTablet for tablet 089e7119ddef4191bcc2c069e80ff066 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:54:47 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:40136
I20260929 10:54:47.155200 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 13003
I20260929 10:54:47.155218 13581 tablet_replica.cc:330] T 089e7119ddef4191bcc2c069e80ff066 P f2a14a2d305643c195076b4d55901271: stopping tablet replica
I20260929 10:54:47.155723 13581 raft_consensus.cc:2229] T 089e7119ddef4191bcc2c069e80ff066 P f2a14a2d305643c195076b4d55901271 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:54:47.155776 13581 raft_consensus.cc:2258] T 089e7119ddef4191bcc2c069e80ff066 P f2a14a2d305643c195076b4d55901271 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:54:47.156042 13581 ts_tablet_manager.cc:1905] T 089e7119ddef4191bcc2c069e80ff066 P f2a14a2d305643c195076b4d55901271: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:54:47.157768 13581 ts_tablet_manager.cc:1918] T 089e7119ddef4191bcc2c069e80ff066 P f2a14a2d305643c195076b4d55901271: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 10:54:47.157850 13581 log.cc:1196] T 089e7119ddef4191bcc2c069e80ff066 P f2a14a2d305643c195076b4d55901271: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-1/wal/wals/089e7119ddef4191bcc2c069e80ff066
I20260929 10:54:47.157932 13580 tablet_replica.cc:330] T 089e7119ddef4191bcc2c069e80ff066 P aeb660dae67845ffafb7f377e67753f3: stopping tablet replica
I20260929 10:54:47.158011 13580 raft_consensus.cc:2229] T 089e7119ddef4191bcc2c069e80ff066 P aeb660dae67845ffafb7f377e67753f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:54:47.158061 13580 raft_consensus.cc:2258] T 089e7119ddef4191bcc2c069e80ff066 P aeb660dae67845ffafb7f377e67753f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:54:47.158139 13581 ts_tablet_manager.cc:1939] T 089e7119ddef4191bcc2c069e80ff066 P f2a14a2d305643c195076b4d55901271: Deleting consensus metadata
I20260929 10:54:47.158357 13580 ts_tablet_manager.cc:1905] T 089e7119ddef4191bcc2c069e80ff066 P aeb660dae67845ffafb7f377e67753f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:54:47.158468 12917 catalog_manager.cc:4841] TS f2a14a2d305643c195076b4d55901271 (127.10.238.194:38251): tablet 089e7119ddef4191bcc2c069e80ff066 (table test-table [id=7f2e8f06cd934f6b892d198d032755ff]) successfully deleted
I20260929 10:54:47.160275 13580 ts_tablet_manager.cc:1918] T 089e7119ddef4191bcc2c069e80ff066 P aeb660dae67845ffafb7f377e67753f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 10:54:47.160338 13580 log.cc:1196] T 089e7119ddef4191bcc2c069e80ff066 P aeb660dae67845ffafb7f377e67753f3: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790679274510869-11195-0/minicluster-data/ts-2/wal/wals/089e7119ddef4191bcc2c069e80ff066
I20260929 10:54:47.160553 13580 ts_tablet_manager.cc:1939] T 089e7119ddef4191bcc2c069e80ff066 P aeb660dae67845ffafb7f377e67753f3: Deleting consensus metadata
I20260929 10:54:47.161034 12916 catalog_manager.cc:4841] TS aeb660dae67845ffafb7f377e67753f3 (127.10.238.195:36733): tablet 089e7119ddef4191bcc2c069e80ff066 (table test-table [id=7f2e8f06cd934f6b892d198d032755ff]) successfully deleted
W20260929 10:54:47.163645 12917 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.238.193:43443 (error 108)
I20260929 10:54:47.163892 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 13143
I20260929 10:54:47.169795 13583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:47.169577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:43443 (local address 127.0.0.1:54970)
0929 10:54:47.169712 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:47.169756 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.193:43443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":81,"threads_started":1}
W20260929 10:54:47.169965 12917 catalog_manager.cc:4584] TS 9ba1f180c87d40bdb61d85c762d7e342 (127.10.238.193:43443): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 089e7119ddef4191bcc2c069e80ff066: Network error: Client connection negotiation failed: client connection to 127.10.238.193:43443: connect: Connection refused (error 111)
I20260929 10:54:47.170476 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 13283
I20260929 10:54:47.176716 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 12899
2026-09-29T10:54:47Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6167 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:54:47 dist-test-slave-8nrt krb5kdc[13586](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:54:47 dist-test-slave-8nrt krb5kdc[13586](info): set up 2 sockets
Sep 29 10:54:47 dist-test-slave-8nrt krb5kdc[13586](info): commencing operation
krb5kdc: starting...
W20260929 10:54:48.209103 11195 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Sep 29 10:54:48 dist-test-slave-8nrt krb5kdc[13586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:54:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:54:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
I20260929 10:54:48.328811 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:44405
--webserver_interface=127.10.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:35371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:44405
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:54:48.408928 13599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:54:48.409157 13599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:54:48.409194 13599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:54:48.410349 13599 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:54:48.410418 13599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:54:48.410444 13599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:54:48.410470 13599 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:54:48.410501 13599 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:54:48.411932 13599 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:35371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:44405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:44405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:54:48.412204 13599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:54:48.412429 13599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:54:48.417500 13605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:48.417506 13604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:48.417536 13599 server_base.cc:1034] running on GCE node
W20260929 10:54:48.417554 13607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:48.417887 13599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:54:48.418221 13599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:54:48.419396 13599 hybrid_clock.cc:648] HybridClock initialized: now 1790679288419393 us; error 42 us; skew 500 ppm
Sep 29 10:54:48 dist-test-slave-8nrt krb5kdc[13586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679288, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:54:48.426362 13599 init.cc:377] Logged in from keytab as kudu/127.10.238.254@KRBTEST.COM (short username kudu)
I20260929 10:54:48.426819 13599 webserver.cc:460] Webserver started at http://127.10.238.254:37483/ using document root <none> and password file <none>
I20260929 10:54:48.427047 13599 fs_manager.cc:359] Metadata directory not provided
I20260929 10:54:48.427122 13599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:54:48.427273 13599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:54:48.428351 13599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/master-0/data/instance:
uuid: "907cbbd9b9ad46718401ed9bc164fb65"
format_stamp: "Formatted at 2026-09-29 10:54:48 on dist-test-slave-8nrt"
server_key: "e7abb0d444b9c18216e50ba98ca23b98"
server_key_iv: "dd5aa64cb0601e12a99b65026510dba0"
server_key_version: "encryptionkey@0"
I20260929 10:54:48.428723 13599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/master-0/wal/instance:
uuid: "907cbbd9b9ad46718401ed9bc164fb65"
format_stamp: "Formatted at 2026-09-29 10:54:48 on dist-test-slave-8nrt"
server_key: "e7abb0d444b9c18216e50ba98ca23b98"
server_key_iv: "dd5aa64cb0601e12a99b65026510dba0"
server_key_version: "encryptionkey@0"
I20260929 10:54:48.430172 13599 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 10:54:48.430886 13614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:48.431051 13599 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 10:54:48.431171 13599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "907cbbd9b9ad46718401ed9bc164fb65"
format_stamp: "Formatted at 2026-09-29 10:54:48 on dist-test-slave-8nrt"
server_key: "e7abb0d444b9c18216e50ba98ca23b98"
server_key_iv: "dd5aa64cb0601e12a99b65026510dba0"
server_key_version: "encryptionkey@0"
I20260929 10:54:48.431248 13599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:54:48.442178 13599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:54:48.444142 13599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:54:48.444270 13599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:54:48.448153 13599 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.254:44405
I20260929 10:54:48.448220 13666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.254:44405 every 8 connection(s)
I20260929 10:54:48.448575 13599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:54:48.449496 13667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:54:48.451741 13667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 907cbbd9b9ad46718401ed9bc164fb65: Bootstrap starting.
I20260929 10:54:48.452330 13667 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 907cbbd9b9ad46718401ed9bc164fb65: Neither blocks nor log segments found. Creating new log.
I20260929 10:54:48.452656 13667 log.cc:824] T 00000000000000000000000000000000 P 907cbbd9b9ad46718401ed9bc164fb65: Log is configured to *not* fsync() on all Append() calls
I20260929 10:54:48.453418 13667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 907cbbd9b9ad46718401ed9bc164fb65: No bootstrap required, opened a new log
I20260929 10:54:48.453469 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 13599
I20260929 10:54:48.453569 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/master-0/wal/instance
I20260929 10:54:48.453766 11195 external_mini_cluster.cc:1428] Setting key cd819afe6e93eba83ccf2183a68811b2
I20260929 10:54:48.455374 13667 raft_consensus.cc:348] T 00000000000000000000000000000000 P 907cbbd9b9ad46718401ed9bc164fb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907cbbd9b9ad46718401ed9bc164fb65" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 44405 } }
I20260929 10:54:48.455529 13667 raft_consensus.cc:374] T 00000000000000000000000000000000 P 907cbbd9b9ad46718401ed9bc164fb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:54:48.455603 13667 raft_consensus.cc:729] T 00000000000000000000000000000000 P 907cbbd9b9ad46718401ed9bc164fb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 907cbbd9b9ad46718401ed9bc164fb65, State: Initialized, Role: FOLLOWER
I20260929 10:54:48.455782 13667 consensus_queue.cc:260] T 00000000000000000000000000000000 P 907cbbd9b9ad46718401ed9bc164fb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907cbbd9b9ad46718401ed9bc164fb65" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 44405 } }
I20260929 10:54:48.455874 13667 raft_consensus.cc:388] T 00000000000000000000000000000000 P 907cbbd9b9ad46718401ed9bc164fb65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:54:48.455921 13667 raft_consensus.cc:482] T 00000000000000000000000000000000 P 907cbbd9b9ad46718401ed9bc164fb65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:54:48.455969 13667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 907cbbd9b9ad46718401ed9bc164fb65 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:54:48.456832 13667 raft_consensus.cc:504] T 00000000000000000000000000000000 P 907cbbd9b9ad46718401ed9bc164fb65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907cbbd9b9ad46718401ed9bc164fb65" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 44405 } }
I20260929 10:54:48.456952 13667 leader_election.cc:302] T 00000000000000000000000000000000 P 907cbbd9b9ad46718401ed9bc164fb65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 907cbbd9b9ad46718401ed9bc164fb65; no voters:
I20260929 10:54:48.457126 13667 leader_election.cc:288] T 00000000000000000000000000000000 P 907cbbd9b9ad46718401ed9bc164fb65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:54:48.457152 13672 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 907cbbd9b9ad46718401ed9bc164fb65 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:54:48.457336 13672 raft_consensus.cc:686] T 00000000000000000000000000000000 P 907cbbd9b9ad46718401ed9bc164fb65 [term 1 LEADER]: Becoming Leader. State: Replica: 907cbbd9b9ad46718401ed9bc164fb65, State: Running, Role: LEADER
I20260929 10:54:48.457446 13667 sys_catalog.cc:564] T 00000000000000000000000000000000 P 907cbbd9b9ad46718401ed9bc164fb65 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:54:48.457531 13672 consensus_queue.cc:237] T 00000000000000000000000000000000 P 907cbbd9b9ad46718401ed9bc164fb65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907cbbd9b9ad46718401ed9bc164fb65" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 44405 } }
I20260929 10:54:48.457904 13672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 907cbbd9b9ad46718401ed9bc164fb65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 907cbbd9b9ad46718401ed9bc164fb65. Latest consensus state: current_term: 1 leader_uuid: "907cbbd9b9ad46718401ed9bc164fb65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907cbbd9b9ad46718401ed9bc164fb65" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 44405 } } }
I20260929 10:54:48.457919 13673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 907cbbd9b9ad46718401ed9bc164fb65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "907cbbd9b9ad46718401ed9bc164fb65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907cbbd9b9ad46718401ed9bc164fb65" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 44405 } } }
I20260929 10:54:48.458175 13673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 907cbbd9b9ad46718401ed9bc164fb65 [sys.catalog]: This master's current role is: LEADER
I20260929 10:54:48.458074 13672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 907cbbd9b9ad46718401ed9bc164fb65 [sys.catalog]: This master's current role is: LEADER
I20260929 10:54:48.458470 13681 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:54:48.459019 13681 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:54:48.460783 13681 catalog_manager.cc:1347] Generated new cluster ID: 794231a6e949440180fe8d4815044e8d
I20260929 10:54:48.460839 13681 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 10:54:48 dist-test-slave-8nrt krb5kdc[13586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:54:48.466466 13671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:48.454277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53572 (local address 127.10.238.254:44405)
0929 10:54:48.454459 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:48.454466 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:48.454491 (+ 25us) server_negotiation.cc:407] Connection header received
0929 10:54:48.455367 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:48.455376 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:48.455433 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:48.455619 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:48.456455 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:48.457644 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:48.458795 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:48.459517 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:48.459650 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:48.462598 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:48.462602 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:48.462603 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:48.462616 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:48.464612 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:48.465028 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:48.465031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:48.465032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:48.465171 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:48.465463 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:48.465464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:48.465464 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:48.465772 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:48.465928 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:48.466110 (+ 182us) server_negotiation.cc:299] Negotiation successful
0929 10:54:48.466269 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":94,"threads_started":1}
I20260929 10:54:48.470081 13681 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:54:48.470649 13681 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:54:48.476955 13681 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 907cbbd9b9ad46718401ed9bc164fb65: Generated new TSK 0
I20260929 10:54:48.477164 13681 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.238.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab.
Entry for principal kudu/127.10.238.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab.
I20260929 10:54:48.504282 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.193:0
--local_ip_for_outbound_sockets=127.10.238.193
--webserver_interface=127.10.238.193
--webserver_port=0
--tserver_master_addrs=127.10.238.254:44405
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab
--principal=kudu/127.10.238.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:35371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:54:48.586900 13693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:54:48.587144 13693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:54:48.587186 13693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:54:48.588375 13693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:54:48.588445 13693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:54:48.588495 13693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.193
I20260929 10:54:48.590030 13693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:35371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab
--principal=kudu/127.10.238.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.238.193
--webserver_port=0
--tserver_master_addrs=127.10.238.254:44405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.193
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:54:48.590314 13693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:54:48.590555 13693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:54:48.595592 13698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:48.595592 13701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:48.595748 13693 server_base.cc:1034] running on GCE node
W20260929 10:54:48.595835 13699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:48.596060 13693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:54:48.596377 13693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:54:48.597604 13693 hybrid_clock.cc:648] HybridClock initialized: now 1790679288597590 us; error 43 us; skew 500 ppm
Sep 29 10:54:48 dist-test-slave-8nrt krb5kdc[13586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679288, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:54:48.599974 13693 init.cc:377] Logged in from keytab as kudu/127.10.238.193@KRBTEST.COM (short username kudu)
I20260929 10:54:48.600437 13693 webserver.cc:460] Webserver started at http://127.10.238.193:45205/ using document root <none> and password file <none>
I20260929 10:54:48.600672 13693 fs_manager.cc:359] Metadata directory not provided
I20260929 10:54:48.600749 13693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:54:48.600903 13693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:54:48.601982 13693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-0/data/instance:
uuid: "cfbf3da4715441228669868d2a6a5058"
format_stamp: "Formatted at 2026-09-29 10:54:48 on dist-test-slave-8nrt"
server_key: "b97375c0e255bc0bbe0c6bfde6091992"
server_key_iv: "7bc6a571a84ece97708dc8732c6e4346"
server_key_version: "encryptionkey@0"
I20260929 10:54:48.602339 13693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-0/wal/instance:
uuid: "cfbf3da4715441228669868d2a6a5058"
format_stamp: "Formatted at 2026-09-29 10:54:48 on dist-test-slave-8nrt"
server_key: "b97375c0e255bc0bbe0c6bfde6091992"
server_key_iv: "7bc6a571a84ece97708dc8732c6e4346"
server_key_version: "encryptionkey@0"
I20260929 10:54:48.603637 13693 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 10:54:48.604358 13708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:48.604528 13693 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:54:48.604625 13693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-0/wal
uuid: "cfbf3da4715441228669868d2a6a5058"
format_stamp: "Formatted at 2026-09-29 10:54:48 on dist-test-slave-8nrt"
server_key: "b97375c0e255bc0bbe0c6bfde6091992"
server_key_iv: "7bc6a571a84ece97708dc8732c6e4346"
server_key_version: "encryptionkey@0"
I20260929 10:54:48.604705 13693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:54:48.611639 13693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:54:48.613382 13693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:54:48.613539 13693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:54:48.613797 13693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:54:48.614092 13693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:54:48.614145 13693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:48.614179 13693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:54:48.614205 13693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:48.620174 13693 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.193:35035
I20260929 10:54:48.620231 13821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.193:35035 every 8 connection(s)
I20260929 10:54:48.620515 13693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
Sep 29 10:54:48 dist-test-slave-8nrt krb5kdc[13586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679288, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.193@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:54:48.629314 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 13693
I20260929 10:54:48.629419 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-0/wal/instance
I20260929 10:54:48.629621 11195 external_mini_cluster.cc:1428] Setting key 93595feac87f9621942641d7cc2333b8
I20260929 10:54:48.629678 13671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:48.621193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:60397 (local address 127.10.238.254:44405)
0929 10:54:48.621298 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:48.621300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:48.621454 (+ 154us) server_negotiation.cc:407] Connection header received
0929 10:54:48.621969 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:48.621970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:48.621983 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:48.622022 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:48.622956 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:48.623452 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:48.624451 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:48.624673 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:48.624763 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:48.627038 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:48.627040 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:48.627040 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:48.627050 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:48.628236 (+ 1186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:48.628642 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:48.628643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:48.628644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:48.628704 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:48.628963 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:48.628964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:48.628964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:48.629079 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:48.629149 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:48.629504 (+ 355us) server_negotiation.cc:299] Negotiation successful
0929 10:54:48.629592 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 10:54:48.629830 13824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:48.621141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:44405 (local address 127.10.238.193:60397)
0929 10:54:48.621325 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:48.621367 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:48.621861 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:48.622096 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:48.622100 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:48.622658 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:54:48.622853 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:48.622858 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:48.623546 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:48.623547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:48.624349 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:48.624351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:48.624775 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:48.624776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:48.624907 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:48.625534 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:54:48.625542 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:54:48.626931 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:54:48.628325 (+ 1394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:48.628333 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:48.628335 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:48.628551 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:48.628776 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:48.628777 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:48.628778 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:48.628888 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:48.629151 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:54:48.629159 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:54:48.629346 (+ 187us) client_negotiation.cc:769] Sending connection context
0929 10:54:48.629463 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 10:54:48.629612 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":89,"threads_started":1}
I20260929 10:54:48.630339 13822 heartbeater.cc:347] Connected to a master server at 127.10.238.254:44405
I20260929 10:54:48.630448 13822 heartbeater.cc:464] Registering TS with master...
I20260929 10:54:48.630620 13822 heartbeater.cc:511] Master 127.10.238.254:44405 requested a full tablet report, sending...
I20260929 10:54:48.631141 13630 ts_manager.cc:194] Registered new tserver with Master: cfbf3da4715441228669868d2a6a5058 (127.10.238.193:35035)
I20260929 10:54:48.631973 13630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.193@KRBTEST.COM'} at 127.10.238.193:60397
WARNING: no policy specified for kudu/127.10.238.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab.
Entry for principal kudu/127.10.238.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab.
I20260929 10:54:48.654034 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.194:0
--local_ip_for_outbound_sockets=127.10.238.194
--webserver_interface=127.10.238.194
--webserver_port=0
--tserver_master_addrs=127.10.238.254:44405
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab
--principal=kudu/127.10.238.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:35371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:54:48.734305 13827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:54:48.734555 13827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:54:48.734594 13827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:54:48.735894 13827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:54:48.735965 13827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:54:48.736012 13827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.194
I20260929 10:54:48.737537 13827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:35371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab
--principal=kudu/127.10.238.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.238.194
--webserver_port=0
--tserver_master_addrs=127.10.238.254:44405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.194
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:54:48.737816 13827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:54:48.738062 13827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:54:48.743165 13833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:48.743139 13835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:48.743257 13827 server_base.cc:1034] running on GCE node
W20260929 10:54:48.743153 13832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:48.743615 13827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:54:48.743916 13827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:54:48.745081 13827 hybrid_clock.cc:648] HybridClock initialized: now 1790679288745075 us; error 35 us; skew 500 ppm
Sep 29 10:54:48 dist-test-slave-8nrt krb5kdc[13586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679288, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:54:48.747521 13827 init.cc:377] Logged in from keytab as kudu/127.10.238.194@KRBTEST.COM (short username kudu)
I20260929 10:54:48.747993 13827 webserver.cc:460] Webserver started at http://127.10.238.194:45537/ using document root <none> and password file <none>
I20260929 10:54:48.748229 13827 fs_manager.cc:359] Metadata directory not provided
I20260929 10:54:48.748301 13827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:54:48.748437 13827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:54:48.749589 13827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-1/data/instance:
uuid: "949b6c86469246d6aaf43edf09631e5d"
format_stamp: "Formatted at 2026-09-29 10:54:48 on dist-test-slave-8nrt"
server_key: "df7571679642de4e301ca5519961a16a"
server_key_iv: "8f6cefa0b68d827f072a4acf7d7e486c"
server_key_version: "encryptionkey@0"
I20260929 10:54:48.749959 13827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-1/wal/instance:
uuid: "949b6c86469246d6aaf43edf09631e5d"
format_stamp: "Formatted at 2026-09-29 10:54:48 on dist-test-slave-8nrt"
server_key: "df7571679642de4e301ca5519961a16a"
server_key_iv: "8f6cefa0b68d827f072a4acf7d7e486c"
server_key_version: "encryptionkey@0"
I20260929 10:54:48.751298 13827 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260929 10:54:48.751996 13842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:48.752166 13827 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:54:48.752271 13827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-1/wal
uuid: "949b6c86469246d6aaf43edf09631e5d"
format_stamp: "Formatted at 2026-09-29 10:54:48 on dist-test-slave-8nrt"
server_key: "df7571679642de4e301ca5519961a16a"
server_key_iv: "8f6cefa0b68d827f072a4acf7d7e486c"
server_key_version: "encryptionkey@0"
I20260929 10:54:48.752353 13827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:54:48.772452 13827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:54:48.774897 13827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:54:48.775068 13827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:54:48.775295 13827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:54:48.775609 13827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:54:48.775651 13827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:48.775673 13827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:54:48.775717 13827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:48.781473 13827 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.194:40411
I20260929 10:54:48.781551 13955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.194:40411 every 8 connection(s)
I20260929 10:54:48.781873 13827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
Sep 29 10:54:48 dist-test-slave-8nrt krb5kdc[13586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679288, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.194@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:54:48.788825 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 13827
I20260929 10:54:48.788929 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-1/wal/instance
I20260929 10:54:48.789203 11195 external_mini_cluster.cc:1428] Setting key f55f5b4dbc68f4641a368f7bb34b8b40
I20260929 10:54:48.792150 13671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:48.782649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:48075 (local address 127.10.238.254:44405)
0929 10:54:48.782730 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:48.782733 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:48.782902 (+ 169us) server_negotiation.cc:407] Connection header received
0929 10:54:48.783442 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:48.783443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:48.783456 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:48.783509 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:48.784448 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:48.784965 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:48.786054 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:48.786269 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:48.786358 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:48.788474 (+ 2116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:48.788476 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:48.788476 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:48.788482 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:48.790305 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:48.790841 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:48.790842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:48.790843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:48.790928 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:48.791244 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:48.791245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:48.791246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:48.791417 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:48.791494 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:48.791897 (+ 403us) server_negotiation.cc:299] Negotiation successful
0929 10:54:48.792022 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 10:54:48.792492 13958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:48.782602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:44405 (local address 127.10.238.194:48075)
0929 10:54:48.782775 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:48.782812 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:48.783324 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:48.783584 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:48.783587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:48.784151 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:54:48.784344 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:48.784349 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:48.785069 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:48.785070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:48.785947 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:48.785949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:48.786368 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:48.786370 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:48.786500 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:48.787059 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:54:48.787067 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:54:48.788364 (+ 1297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:54:48.790403 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:48.790411 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:48.790413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:48.790758 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:48.791000 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:48.791001 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:48.791002 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:48.791186 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:48.791494 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:54:48.791500 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:54:48.791710 (+ 210us) client_negotiation.cc:769] Sending connection context
0929 10:54:48.791831 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 10:54:48.792242 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":102,"threads_started":1}
I20260929 10:54:48.793009 13956 heartbeater.cc:347] Connected to a master server at 127.10.238.254:44405
I20260929 10:54:48.793128 13956 heartbeater.cc:464] Registering TS with master...
I20260929 10:54:48.793355 13956 heartbeater.cc:511] Master 127.10.238.254:44405 requested a full tablet report, sending...
I20260929 10:54:48.793793 13630 ts_manager.cc:194] Registered new tserver with Master: 949b6c86469246d6aaf43edf09631e5d (127.10.238.194:40411)
I20260929 10:54:48.794286 13630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.194@KRBTEST.COM'} at 127.10.238.194:48075
WARNING: no policy specified for kudu/127.10.238.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab.
Entry for principal kudu/127.10.238.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab.
I20260929 10:54:48.817639 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.195:0
--local_ip_for_outbound_sockets=127.10.238.195
--webserver_interface=127.10.238.195
--webserver_port=0
--tserver_master_addrs=127.10.238.254:44405
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab
--principal=kudu/127.10.238.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:35371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:54:48.900954 13961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:54:48.901168 13961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:54:48.901192 13961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:54:48.902459 13961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:54:48.902500 13961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:54:48.902526 13961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.195
I20260929 10:54:48.903957 13961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:35371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab
--principal=kudu/127.10.238.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.238.195
--webserver_port=0
--tserver_master_addrs=127.10.238.254:44405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.195
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:54:48.904198 13961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:54:48.904448 13961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:54:48.909504 13966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:48.909515 13967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:48.909571 13969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:48.909745 13961 server_base.cc:1034] running on GCE node
I20260929 10:54:48.909958 13961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:54:48.910192 13961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:54:48.911360 13961 hybrid_clock.cc:648] HybridClock initialized: now 1790679288911339 us; error 33 us; skew 500 ppm
Sep 29 10:54:48 dist-test-slave-8nrt krb5kdc[13586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679288, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:54:48.913818 13961 init.cc:377] Logged in from keytab as kudu/127.10.238.195@KRBTEST.COM (short username kudu)
I20260929 10:54:48.914288 13961 webserver.cc:460] Webserver started at http://127.10.238.195:39001/ using document root <none> and password file <none>
I20260929 10:54:48.914518 13961 fs_manager.cc:359] Metadata directory not provided
I20260929 10:54:48.914585 13961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:54:48.914724 13961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:54:48.915760 13961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-2/data/instance:
uuid: "9e02493c7a334deb94950683b59c0cf8"
format_stamp: "Formatted at 2026-09-29 10:54:48 on dist-test-slave-8nrt"
server_key: "eb450644c691e76be1b7dc9f9da39068"
server_key_iv: "6f444415081747cd1c635eb077f063e9"
server_key_version: "encryptionkey@0"
I20260929 10:54:48.916126 13961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-2/wal/instance:
uuid: "9e02493c7a334deb94950683b59c0cf8"
format_stamp: "Formatted at 2026-09-29 10:54:48 on dist-test-slave-8nrt"
server_key: "eb450644c691e76be1b7dc9f9da39068"
server_key_iv: "6f444415081747cd1c635eb077f063e9"
server_key_version: "encryptionkey@0"
I20260929 10:54:48.917533 13961 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 10:54:48.918283 13976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:48.918467 13961 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:54:48.918563 13961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-2/wal
uuid: "9e02493c7a334deb94950683b59c0cf8"
format_stamp: "Formatted at 2026-09-29 10:54:48 on dist-test-slave-8nrt"
server_key: "eb450644c691e76be1b7dc9f9da39068"
server_key_iv: "6f444415081747cd1c635eb077f063e9"
server_key_version: "encryptionkey@0"
I20260929 10:54:48.918648 13961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:54:48.941463 13961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:54:48.943378 13961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:54:48.943555 13961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:54:48.943823 13961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:54:48.944150 13961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:54:48.944204 13961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:48.944242 13961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:54:48.944269 13961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:48.950165 13961 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.195:37375
I20260929 10:54:48.950239 14089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.195:37375 every 8 connection(s)
I20260929 10:54:48.950565 13961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
I20260929 10:54:48.952669 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 13961
I20260929 10:54:48.952793 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-2/wal/instance
I20260929 10:54:48.952975 11195 external_mini_cluster.cc:1428] Setting key c16f2c6eecbbcd41cb9df6b5b789ba42
Sep 29 10:54:48 dist-test-slave-8nrt krb5kdc[13586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679288, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.195@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:54:48.960682 13671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:48.951320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:54447 (local address 127.10.238.254:44405)
0929 10:54:48.951388 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:48.951392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:48.951605 (+ 213us) server_negotiation.cc:407] Connection header received
0929 10:54:48.952131 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:48.952132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:48.952151 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:48.952218 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:48.953346 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:48.953865 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:48.954997 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:48.955218 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:48.955297 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:48.957706 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:48.957707 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:48.957707 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:48.957714 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:48.959427 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:48.959795 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:48.959796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:48.959797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:48.959856 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:48.960122 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:48.960123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:48.960123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:48.960235 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:48.960291 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:48.960539 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 10:54:48.960616 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 10:54:48.960876 14092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:48.951302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:44405 (local address 127.10.238.195:54447)
0929 10:54:48.951484 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:48.951520 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:48.951992 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:48.952310 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:48.952313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:48.952948 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:54:48.953226 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:48.953233 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:48.953997 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:48.953998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:48.954893 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:48.954895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:48.955306 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:48.955308 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:48.955433 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:48.955995 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:54:48.956003 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:54:48.957620 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:54:48.959503 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:48.959508 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:48.959510 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:48.959726 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:48.959943 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:48.959944 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:48.959944 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:48.960056 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:48.960295 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:54:48.960299 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:54:48.960411 (+ 112us) client_negotiation.cc:769] Sending connection context
0929 10:54:48.960502 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 10:54:48.960634 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":103,"threads_started":1}
I20260929 10:54:48.961535 14090 heartbeater.cc:347] Connected to a master server at 127.10.238.254:44405
I20260929 10:54:48.961680 14090 heartbeater.cc:464] Registering TS with master...
I20260929 10:54:48.961871 14090 heartbeater.cc:511] Master 127.10.238.254:44405 requested a full tablet report, sending...
I20260929 10:54:48.962260 13630 ts_manager.cc:194] Registered new tserver with Master: 9e02493c7a334deb94950683b59c0cf8 (127.10.238.195:37375)
I20260929 10:54:48.962637 13630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.195@KRBTEST.COM'} at 127.10.238.195:54447
I20260929 10:54:48.964597 11195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 10:54:48.975091 13671 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:54:48.974214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53586 (local address 127.10.238.254:44405)
0929 10:54:48.974277 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:48.974278 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:48.974418 (+ 140us) server_negotiation.cc:407] Connection header received
0929 10:54:48.974461 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:48.974461 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:48.974471 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:48.974520 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:48.975023 (+ 503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53586: BlockingRecv error: recv got EOF from 127.0.0.1:53586 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 10:54:48.975911 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 13693
I20260929 10:54:48.980443 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 13827
I20260929 10:54:48.984490 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 13961
I20260929 10:54:48.988888 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 13599
2026-09-29T10:54:48Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1810 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790679274510869-11195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:54:49 dist-test-slave-8nrt krb5kdc[14102](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:54:49 dist-test-slave-8nrt krb5kdc[14102](info): set up 2 sockets
Sep 29 10:54:49 dist-test-slave-8nrt krb5kdc[14102](info): commencing operation
krb5kdc: starting...
W20260929 10:54:50.069206 11195 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Sep 29 10:54:50 dist-test-slave-8nrt krb5kdc[14102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:54:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:54:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
I20260929 10:54:50.184849 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:44745
--webserver_interface=127.10.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:40089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:44745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:54:50.263521 14115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:54:50.263756 14115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:54:50.263792 14115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:54:50.265033 14115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:54:50.265100 14115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:54:50.265125 14115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:54:50.265157 14115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:54:50.265183 14115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:54:50.266739 14115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:40089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790679274510869-11195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:44745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:44745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:54:50.267004 14115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:54:50.267273 14115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:54:50.272171 14120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:50.272259 14121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:50.272306 14115 server_base.cc:1034] running on GCE node
W20260929 10:54:50.272320 14123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:50.272627 14115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:54:50.272920 14115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:54:50.274081 14115 hybrid_clock.cc:648] HybridClock initialized: now 1790679290274070 us; error 46 us; skew 500 ppm
Sep 29 10:54:50 dist-test-slave-8nrt krb5kdc[14102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679290, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:54:50.276479 14115 init.cc:377] Logged in from keytab as kudu/127.10.238.254@KRBTEST.COM (short username kudu)
I20260929 10:54:50.276923 14115 webserver.cc:460] Webserver started at http://127.10.238.254:36701/ using document root <none> and password file <none>
I20260929 10:54:50.277099 14115 fs_manager.cc:359] Metadata directory not provided
I20260929 10:54:50.277138 14115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:54:50.277225 14115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:54:50.278294 14115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790679274510869-11195-0/minicluster-data/master-0/data/instance:
uuid: "d27888a9e9f4467a88cff32ab57affb6"
format_stamp: "Formatted at 2026-09-29 10:54:50 on dist-test-slave-8nrt"
server_key: "c1059946ccc344e12d38ddc0f8cf474d"
server_key_iv: "21ed5c5bed3b3bb2e557a2b6a8c9eeff"
server_key_version: "encryptionkey@0"
I20260929 10:54:50.278599 14115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790679274510869-11195-0/minicluster-data/master-0/wal/instance:
uuid: "d27888a9e9f4467a88cff32ab57affb6"
format_stamp: "Formatted at 2026-09-29 10:54:50 on dist-test-slave-8nrt"
server_key: "c1059946ccc344e12d38ddc0f8cf474d"
server_key_iv: "21ed5c5bed3b3bb2e557a2b6a8c9eeff"
server_key_version: "encryptionkey@0"
I20260929 10:54:50.279830 14115 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 10:54:50.280484 14130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:50.280673 14115 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 10:54:50.280762 14115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "d27888a9e9f4467a88cff32ab57affb6"
format_stamp: "Formatted at 2026-09-29 10:54:50 on dist-test-slave-8nrt"
server_key: "c1059946ccc344e12d38ddc0f8cf474d"
server_key_iv: "21ed5c5bed3b3bb2e557a2b6a8c9eeff"
server_key_version: "encryptionkey@0"
I20260929 10:54:50.280836 14115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:54:50.311128 14115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:54:50.313201 14115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:54:50.313393 14115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:54:50.317322 14115 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.254:44745
I20260929 10:54:50.317395 14182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.254:44745 every 8 connection(s)
I20260929 10:54:50.317873 14115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:54:50.318523 14183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:54:50.319840 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 14115
I20260929 10:54:50.319945 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790679274510869-11195-0/minicluster-data/master-0/wal/instance
I20260929 10:54:50.320135 11195 external_mini_cluster.cc:1428] Setting key eb2fb36ce6e96ecb0712f7ead2e56d67
I20260929 10:54:50.321249 14183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d27888a9e9f4467a88cff32ab57affb6: Bootstrap starting.
I20260929 10:54:50.321911 14183 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d27888a9e9f4467a88cff32ab57affb6: Neither blocks nor log segments found. Creating new log.
I20260929 10:54:50.322214 14183 log.cc:824] T 00000000000000000000000000000000 P d27888a9e9f4467a88cff32ab57affb6: Log is configured to *not* fsync() on all Append() calls
I20260929 10:54:50.322892 14183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d27888a9e9f4467a88cff32ab57affb6: No bootstrap required, opened a new log
I20260929 10:54:50.324616 14183 raft_consensus.cc:348] T 00000000000000000000000000000000 P d27888a9e9f4467a88cff32ab57affb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d27888a9e9f4467a88cff32ab57affb6" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 44745 } }
I20260929 10:54:50.324790 14183 raft_consensus.cc:374] T 00000000000000000000000000000000 P d27888a9e9f4467a88cff32ab57affb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:54:50.324831 14183 raft_consensus.cc:729] T 00000000000000000000000000000000 P d27888a9e9f4467a88cff32ab57affb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d27888a9e9f4467a88cff32ab57affb6, State: Initialized, Role: FOLLOWER
I20260929 10:54:50.324954 14183 consensus_queue.cc:260] T 00000000000000000000000000000000 P d27888a9e9f4467a88cff32ab57affb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d27888a9e9f4467a88cff32ab57affb6" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 44745 } }
I20260929 10:54:50.325039 14183 raft_consensus.cc:388] T 00000000000000000000000000000000 P d27888a9e9f4467a88cff32ab57affb6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:54:50.325081 14183 raft_consensus.cc:482] T 00000000000000000000000000000000 P d27888a9e9f4467a88cff32ab57affb6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:54:50.325130 14183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d27888a9e9f4467a88cff32ab57affb6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:54:50.325677 14183 raft_consensus.cc:504] T 00000000000000000000000000000000 P d27888a9e9f4467a88cff32ab57affb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d27888a9e9f4467a88cff32ab57affb6" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 44745 } }
I20260929 10:54:50.325768 14183 leader_election.cc:302] T 00000000000000000000000000000000 P d27888a9e9f4467a88cff32ab57affb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d27888a9e9f4467a88cff32ab57affb6; no voters:
I20260929 10:54:50.325930 14183 leader_election.cc:288] T 00000000000000000000000000000000 P d27888a9e9f4467a88cff32ab57affb6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:54:50.325970 14188 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d27888a9e9f4467a88cff32ab57affb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:54:50.326117 14188 raft_consensus.cc:686] T 00000000000000000000000000000000 P d27888a9e9f4467a88cff32ab57affb6 [term 1 LEADER]: Becoming Leader. State: Replica: d27888a9e9f4467a88cff32ab57affb6, State: Running, Role: LEADER
I20260929 10:54:50.326197 14183 sys_catalog.cc:564] T 00000000000000000000000000000000 P d27888a9e9f4467a88cff32ab57affb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:54:50.326216 14188 consensus_queue.cc:237] T 00000000000000000000000000000000 P d27888a9e9f4467a88cff32ab57affb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d27888a9e9f4467a88cff32ab57affb6" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 44745 } }
I20260929 10:54:50.326663 14189 sys_catalog.cc:455] T 00000000000000000000000000000000 P d27888a9e9f4467a88cff32ab57affb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d27888a9e9f4467a88cff32ab57affb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d27888a9e9f4467a88cff32ab57affb6" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 44745 } } }
I20260929 10:54:50.326761 14189 sys_catalog.cc:458] T 00000000000000000000000000000000 P d27888a9e9f4467a88cff32ab57affb6 [sys.catalog]: This master's current role is: LEADER
Sep 29 10:54:50 dist-test-slave-8nrt krb5kdc[14102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:54:50.326941 14190 sys_catalog.cc:455] T 00000000000000000000000000000000 P d27888a9e9f4467a88cff32ab57affb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d27888a9e9f4467a88cff32ab57affb6. Latest consensus state: current_term: 1 leader_uuid: "d27888a9e9f4467a88cff32ab57affb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d27888a9e9f4467a88cff32ab57affb6" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 44745 } } }
I20260929 10:54:50.327013 14190 sys_catalog.cc:458] T 00000000000000000000000000000000 P d27888a9e9f4467a88cff32ab57affb6 [sys.catalog]: This master's current role is: LEADER
I20260929 10:54:50.327167 14196 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:54:50.327668 14196 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:54:50.329452 14196 catalog_manager.cc:1347] Generated new cluster ID: c4e6df52aec44961b7c6be1daecb234b
I20260929 10:54:50.329504 14196 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:54:50.331908 14186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:50.320634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38402 (local address 127.10.238.254:44745)
0929 10:54:50.320826 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:50.320835 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:50.320866 (+ 31us) server_negotiation.cc:407] Connection header received
0929 10:54:50.321333 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:50.321349 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:50.321427 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:50.321583 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:50.322283 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:50.323281 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:50.324010 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:50.324429 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:50.324542 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:50.327472 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:50.327477 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:50.327477 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:50.327489 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:50.330257 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:50.330660 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:50.330663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:50.330663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:50.330774 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:50.331003 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:50.331004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:50.331004 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:50.331292 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:50.331426 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:50.331569 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 10:54:50.331697 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":80,"threads_started":1}
I20260929 10:54:50.332892 11195 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:54:50.347267 14196 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:54:50.347802 14196 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:54:50.353149 14196 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d27888a9e9f4467a88cff32ab57affb6: Generated new TSK 0
I20260929 10:54:50.353354 14196 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:54:50.354949 14186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:50.350884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38412 (local address 127.10.238.254:44745)
0929 10:54:50.350927 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:50.350930 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:50.351013 (+ 83us) server_negotiation.cc:407] Connection header received
0929 10:54:50.351062 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:50.351063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:50.351074 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:50.351150 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:50.351547 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:50.352374 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:50.353385 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:50.353695 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:50.353769 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:50.354470 (+ 701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:50.354471 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:50.354472 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:54:50.354680 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:50.354697 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:54:50.354704 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:50.354789 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:50.354844 (+ 55us) server_negotiation.cc:299] Negotiation successful
0929 10:54:50.354871 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 10:54:50.357198 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 14115
2026-09-29T10:54:50Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1365 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:54:50 dist-test-slave-8nrt krb5kdc[14216](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:54:50 dist-test-slave-8nrt krb5kdc[14216](info): set up 2 sockets
Sep 29 10:54:50 dist-test-slave-8nrt krb5kdc[14216](info): commencing operation
krb5kdc: starting...
W20260929 10:54:51.381075 11195 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Sep 29 10:54:51 dist-test-slave-8nrt krb5kdc[14216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:54:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:54:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
I20260929 10:54:51.497970 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:41611
--webserver_interface=127.10.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:45027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:41611
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:54:51.578162 14229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:54:51.578369 14229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:54:51.578392 14229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:54:51.579483 14229 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:54:51.579516 14229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:54:51.579531 14229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:54:51.579543 14229 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:54:51.579555 14229 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:54:51.580929 14229 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:45027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:41611
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:41611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:54:51.581130 14229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:54:51.581362 14229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:54:51.586081 14234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:51.586081 14237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:51.586139 14235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:51.586208 14229 server_base.cc:1034] running on GCE node
I20260929 10:54:51.586524 14229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:54:51.586802 14229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:54:51.587968 14229 hybrid_clock.cc:648] HybridClock initialized: now 1790679291587956 us; error 27 us; skew 500 ppm
Sep 29 10:54:51 dist-test-slave-8nrt krb5kdc[14216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679291, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:54:51.590561 14229 init.cc:377] Logged in from keytab as kudu/127.10.238.254@KRBTEST.COM (short username kudu)
I20260929 10:54:51.590924 14229 webserver.cc:460] Webserver started at http://127.10.238.254:37073/ using document root <none> and password file <none>
I20260929 10:54:51.591114 14229 fs_manager.cc:359] Metadata directory not provided
I20260929 10:54:51.591154 14229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:54:51.591259 14229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:54:51.592842 14229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/master-0/data/instance:
uuid: "a0870dc8fe60487681cdb57ca01fd330"
format_stamp: "Formatted at 2026-09-29 10:54:51 on dist-test-slave-8nrt"
server_key: "b71d336e991fb484874506c94c2665c7"
server_key_iv: "649b204ba763e52f295623561bfa643d"
server_key_version: "encryptionkey@0"
I20260929 10:54:51.593367 14229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/master-0/wal/instance:
uuid: "a0870dc8fe60487681cdb57ca01fd330"
format_stamp: "Formatted at 2026-09-29 10:54:51 on dist-test-slave-8nrt"
server_key: "b71d336e991fb484874506c94c2665c7"
server_key_iv: "649b204ba763e52f295623561bfa643d"
server_key_version: "encryptionkey@0"
I20260929 10:54:51.595611 14229 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260929 10:54:51.596673 14244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:51.596855 14229 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:54:51.596949 14229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "a0870dc8fe60487681cdb57ca01fd330"
format_stamp: "Formatted at 2026-09-29 10:54:51 on dist-test-slave-8nrt"
server_key: "b71d336e991fb484874506c94c2665c7"
server_key_iv: "649b204ba763e52f295623561bfa643d"
server_key_version: "encryptionkey@0"
I20260929 10:54:51.597069 14229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:54:51.634882 14229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:54:51.637387 14229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:54:51.637552 14229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:54:51.641469 14229 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.254:41611
I20260929 10:54:51.641533 14296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.254:41611 every 8 connection(s)
I20260929 10:54:51.641917 14229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:54:51.642572 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 14229
I20260929 10:54:51.642696 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/master-0/wal/instance
I20260929 10:54:51.642889 11195 external_mini_cluster.cc:1428] Setting key 9d371944b3359eaead6f2ce3660c4fed
I20260929 10:54:51.642979 14297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:54:51.646301 14297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0870dc8fe60487681cdb57ca01fd330: Bootstrap starting.
I20260929 10:54:51.646989 14297 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0870dc8fe60487681cdb57ca01fd330: Neither blocks nor log segments found. Creating new log.
I20260929 10:54:51.647318 14297 log.cc:824] T 00000000000000000000000000000000 P a0870dc8fe60487681cdb57ca01fd330: Log is configured to *not* fsync() on all Append() calls
I20260929 10:54:51.648032 14297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0870dc8fe60487681cdb57ca01fd330: No bootstrap required, opened a new log
Sep 29 10:54:51 dist-test-slave-8nrt krb5kdc[14216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:54:51.650305 14297 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0870dc8fe60487681cdb57ca01fd330 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0870dc8fe60487681cdb57ca01fd330" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 41611 } }
I20260929 10:54:51.650484 14297 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0870dc8fe60487681cdb57ca01fd330 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:54:51.650524 14297 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0870dc8fe60487681cdb57ca01fd330 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0870dc8fe60487681cdb57ca01fd330, State: Initialized, Role: FOLLOWER
I20260929 10:54:51.650666 14297 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0870dc8fe60487681cdb57ca01fd330 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0870dc8fe60487681cdb57ca01fd330" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 41611 } }
I20260929 10:54:51.650749 14297 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0870dc8fe60487681cdb57ca01fd330 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:54:51.650772 14297 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0870dc8fe60487681cdb57ca01fd330 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:54:51.650848 14297 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0870dc8fe60487681cdb57ca01fd330 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:54:51.651530 14297 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0870dc8fe60487681cdb57ca01fd330 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0870dc8fe60487681cdb57ca01fd330" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 41611 } }
I20260929 10:54:51.651625 14297 leader_election.cc:302] T 00000000000000000000000000000000 P a0870dc8fe60487681cdb57ca01fd330 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0870dc8fe60487681cdb57ca01fd330; no voters:
I20260929 10:54:51.651836 14297 leader_election.cc:288] T 00000000000000000000000000000000 P a0870dc8fe60487681cdb57ca01fd330 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:54:51.651925 14302 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0870dc8fe60487681cdb57ca01fd330 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:54:51.652113 14297 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0870dc8fe60487681cdb57ca01fd330 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:54:51.652139 14302 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0870dc8fe60487681cdb57ca01fd330 [term 1 LEADER]: Becoming Leader. State: Replica: a0870dc8fe60487681cdb57ca01fd330, State: Running, Role: LEADER
I20260929 10:54:51.652316 14302 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0870dc8fe60487681cdb57ca01fd330 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0870dc8fe60487681cdb57ca01fd330" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 41611 } }
I20260929 10:54:51.652823 14303 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0870dc8fe60487681cdb57ca01fd330 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0870dc8fe60487681cdb57ca01fd330" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0870dc8fe60487681cdb57ca01fd330" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 41611 } } }
I20260929 10:54:51.652909 14303 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0870dc8fe60487681cdb57ca01fd330 [sys.catalog]: This master's current role is: LEADER
I20260929 10:54:51.653599 14304 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0870dc8fe60487681cdb57ca01fd330 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0870dc8fe60487681cdb57ca01fd330. Latest consensus state: current_term: 1 leader_uuid: "a0870dc8fe60487681cdb57ca01fd330" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0870dc8fe60487681cdb57ca01fd330" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 41611 } } }
I20260929 10:54:51.653676 14304 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0870dc8fe60487681cdb57ca01fd330 [sys.catalog]: This master's current role is: LEADER
I20260929 10:54:51.654143 14300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:51.643404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43050 (local address 127.10.238.254:41611)
0929 10:54:51.643571 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:51.643579 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:51.643602 (+ 23us) server_negotiation.cc:407] Connection header received
0929 10:54:51.644075 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:51.644080 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:51.644125 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:51.644231 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:51.644987 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:51.645966 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:51.646980 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:51.647444 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:51.647551 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:51.650178 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:51.650181 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:51.650182 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:51.650192 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:51.652230 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:51.652642 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:51.652645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:51.652646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:51.652749 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:51.652949 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:51.652950 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:51.652950 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:51.653205 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:51.653635 (+ 430us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:51.653749 (+ 114us) server_negotiation.cc:299] Negotiation successful
0929 10:54:51.653920 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":83,"threads_started":1}
W20260929 10:54:51.654862 14317 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a0870dc8fe60487681cdb57ca01fd330: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:54:51.654915 14317 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:54:51.654982 14311 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:54:51.655130 14311 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:54:51.656466 14311 catalog_manager.cc:1347] Generated new cluster ID: ae7a043dfc7246fabc3fa91a9dcd9c0a
I20260929 10:54:51.656514 14311 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:54:51.665338 14311 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:54:51.665863 14311 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:54:51.673851 14311 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0870dc8fe60487681cdb57ca01fd330: Generated new TSK 0
I20260929 10:54:51.674031 14311 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.238.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab.
Entry for principal kudu/127.10.238.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab.
I20260929 10:54:51.707569 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.193:0
--local_ip_for_outbound_sockets=127.10.238.193
--webserver_interface=127.10.238.193
--webserver_port=0
--tserver_master_addrs=127.10.238.254:41611
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab
--principal=kudu/127.10.238.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:45027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:54:51.787968 14323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:54:51.788158 14323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:54:51.788180 14323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:54:51.789439 14323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:54:51.789479 14323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:54:51.789525 14323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.193
I20260929 10:54:51.790918 14323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:45027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab
--principal=kudu/127.10.238.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.238.193
--webserver_port=0
--tserver_master_addrs=127.10.238.254:41611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.193
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:54:51.791164 14323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:54:51.791425 14323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:54:51.796314 14329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:51.796342 14328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:51.796383 14323 server_base.cc:1034] running on GCE node
W20260929 10:54:51.796422 14331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:51.796756 14323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:54:51.796996 14323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:54:51.798171 14323 hybrid_clock.cc:648] HybridClock initialized: now 1790679291798162 us; error 39 us; skew 500 ppm
Sep 29 10:54:51 dist-test-slave-8nrt krb5kdc[14216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679291, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:54:51.800529 14323 init.cc:377] Logged in from keytab as kudu/127.10.238.193@KRBTEST.COM (short username kudu)
I20260929 10:54:51.801002 14323 webserver.cc:460] Webserver started at http://127.10.238.193:46337/ using document root <none> and password file <none>
I20260929 10:54:51.801225 14323 fs_manager.cc:359] Metadata directory not provided
I20260929 10:54:51.801331 14323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:54:51.801487 14323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:54:51.802570 14323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-0/data/instance:
uuid: "e7cf2299b18f4caa93b47277dd6b362e"
format_stamp: "Formatted at 2026-09-29 10:54:51 on dist-test-slave-8nrt"
server_key: "b49e9b8c52eb34d9895171df85e9e769"
server_key_iv: "8c3d3c20a2b40aa74a5b30f9158bb6db"
server_key_version: "encryptionkey@0"
I20260929 10:54:51.802924 14323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-0/wal/instance:
uuid: "e7cf2299b18f4caa93b47277dd6b362e"
format_stamp: "Formatted at 2026-09-29 10:54:51 on dist-test-slave-8nrt"
server_key: "b49e9b8c52eb34d9895171df85e9e769"
server_key_iv: "8c3d3c20a2b40aa74a5b30f9158bb6db"
server_key_version: "encryptionkey@0"
I20260929 10:54:51.804194 14323 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 10:54:51.804917 14338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:51.805114 14323 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:54:51.805220 14323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-0/wal
uuid: "e7cf2299b18f4caa93b47277dd6b362e"
format_stamp: "Formatted at 2026-09-29 10:54:51 on dist-test-slave-8nrt"
server_key: "b49e9b8c52eb34d9895171df85e9e769"
server_key_iv: "8c3d3c20a2b40aa74a5b30f9158bb6db"
server_key_version: "encryptionkey@0"
I20260929 10:54:51.805320 14323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:54:51.813122 14323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:54:51.814813 14323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:54:51.814966 14323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:54:51.815176 14323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:54:51.815491 14323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:54:51.815557 14323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:51.815613 14323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:54:51.815639 14323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:51.821434 14323 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.193:36337
I20260929 10:54:51.821473 14451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.193:36337 every 8 connection(s)
I20260929 10:54:51.821777 14323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
I20260929 10:54:51.822111 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 14323
I20260929 10:54:51.822223 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-0/wal/instance
I20260929 10:54:51.822423 11195 external_mini_cluster.cc:1428] Setting key 9eb4b1a678c11ef3a37b5bf5afc3cd43
Sep 29 10:54:51 dist-test-slave-8nrt krb5kdc[14216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679291, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.193@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.238.194@KRBTEST.COM; defaulting to no policy
I20260929 10:54:51.831290 14300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:51.822548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:52009 (local address 127.10.238.254:41611)
0929 10:54:51.822605 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:51.822608 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:51.822893 (+ 285us) server_negotiation.cc:407] Connection header received
0929 10:54:51.823473 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:51.823475 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:51.823487 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:51.823557 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:51.824506 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:51.825009 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:51.826072 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:51.826303 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:51.826389 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:51.828458 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:51.828459 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:51.828460 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:51.828467 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:51.829890 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:51.830324 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:51.830326 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:51.830326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:51.830387 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:51.830634 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:51.830635 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:51.830635 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:51.830750 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:51.830824 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:51.831096 (+ 272us) server_negotiation.cc:299] Negotiation successful
0929 10:54:51.831183 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 10:54:51.831442 14454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:51.822515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:41611 (local address 127.10.238.193:52009)
0929 10:54:51.822779 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:51.822825 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:51.823373 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:51.823606 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:51.823610 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:51.824222 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:54:51.824408 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:51.824412 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:51.825108 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:51.825109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:51.825975 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:51.825978 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:51.826402 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:51.826403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:51.826521 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:51.827050 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:54:51.827058 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:54:51.828359 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:54:51.829988 (+ 1629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:51.829995 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:51.829997 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:51.830233 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:51.830454 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:51.830455 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:51.830455 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:51.830565 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:51.830830 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:54:51.830837 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:54:51.830968 (+ 131us) client_negotiation.cc:769] Sending connection context
0929 10:54:51.831074 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 10:54:51.831222 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":77,"threads_started":1}
I20260929 10:54:51.831976 14452 heartbeater.cc:347] Connected to a master server at 127.10.238.254:41611
I20260929 10:54:51.832077 14452 heartbeater.cc:464] Registering TS with master...
I20260929 10:54:51.832249 14452 heartbeater.cc:511] Master 127.10.238.254:41611 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.194@KRBTEST.COM" created.
I20260929 10:54:51.832821 14261 ts_manager.cc:194] Registered new tserver with Master: e7cf2299b18f4caa93b47277dd6b362e (127.10.238.193:36337)
I20260929 10:54:51.833906 14261 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.193@KRBTEST.COM'} at 127.10.238.193:52009
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab.
Entry for principal kudu/127.10.238.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab.
I20260929 10:54:51.846232 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.194:0
--local_ip_for_outbound_sockets=127.10.238.194
--webserver_interface=127.10.238.194
--webserver_port=0
--tserver_master_addrs=127.10.238.254:41611
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab
--principal=kudu/127.10.238.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:45027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:54:51.928429 14457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:54:51.928620 14457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:54:51.928643 14457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:54:51.929747 14457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:54:51.929781 14457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:54:51.929842 14457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.194
I20260929 10:54:51.931308 14457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:45027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab
--principal=kudu/127.10.238.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.238.194
--webserver_port=0
--tserver_master_addrs=127.10.238.254:41611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.194
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:54:51.931486 14457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:54:51.931675 14457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:54:51.936506 14462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:51.936494 14465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:51.936569 14457 server_base.cc:1034] running on GCE node
W20260929 10:54:51.936496 14463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:51.936868 14457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:54:51.937058 14457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:54:51.938190 14457 hybrid_clock.cc:648] HybridClock initialized: now 1790679291938175 us; error 32 us; skew 500 ppm
Sep 29 10:54:51 dist-test-slave-8nrt krb5kdc[14216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679291, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:54:51.940560 14457 init.cc:377] Logged in from keytab as kudu/127.10.238.194@KRBTEST.COM (short username kudu)
I20260929 10:54:51.940972 14457 webserver.cc:460] Webserver started at http://127.10.238.194:38283/ using document root <none> and password file <none>
I20260929 10:54:51.941193 14457 fs_manager.cc:359] Metadata directory not provided
I20260929 10:54:51.941238 14457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:54:51.941401 14457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:54:51.942425 14457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-1/data/instance:
uuid: "75aae7dfcd4b4e4db94f7a674f1e4def"
format_stamp: "Formatted at 2026-09-29 10:54:51 on dist-test-slave-8nrt"
server_key: "78265536e9be8221d76f1c7d77922ea2"
server_key_iv: "8eebc01e36a9973726e348bb1d07d8c4"
server_key_version: "encryptionkey@0"
I20260929 10:54:51.942776 14457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-1/wal/instance:
uuid: "75aae7dfcd4b4e4db94f7a674f1e4def"
format_stamp: "Formatted at 2026-09-29 10:54:51 on dist-test-slave-8nrt"
server_key: "78265536e9be8221d76f1c7d77922ea2"
server_key_iv: "8eebc01e36a9973726e348bb1d07d8c4"
server_key_version: "encryptionkey@0"
I20260929 10:54:51.944027 14457 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 10:54:51.944726 14472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:51.944913 14457 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:54:51.945016 14457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-1/wal
uuid: "75aae7dfcd4b4e4db94f7a674f1e4def"
format_stamp: "Formatted at 2026-09-29 10:54:51 on dist-test-slave-8nrt"
server_key: "78265536e9be8221d76f1c7d77922ea2"
server_key_iv: "8eebc01e36a9973726e348bb1d07d8c4"
server_key_version: "encryptionkey@0"
I20260929 10:54:51.945088 14457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:54:51.958236 14457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:54:51.960114 14457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:54:51.960269 14457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:54:51.960450 14457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:54:51.960749 14457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:54:51.960780 14457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:51.960800 14457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:54:51.960813 14457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:51.966754 14457 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.194:37155
I20260929 10:54:51.966797 14585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.194:37155 every 8 connection(s)
I20260929 10:54:51.967110 14457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
I20260929 10:54:51.970527 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 14457
I20260929 10:54:51.970606 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-1/wal/instance
I20260929 10:54:51.970754 11195 external_mini_cluster.cc:1428] Setting key 520c7f1cc394a80bfd4536575db80488
Sep 29 10:54:51 dist-test-slave-8nrt krb5kdc[14216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679291, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.194@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:54:51.977662 14300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:51.967872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:55927 (local address 127.10.238.254:41611)
0929 10:54:51.967937 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:51.967939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:51.968137 (+ 198us) server_negotiation.cc:407] Connection header received
0929 10:54:51.968670 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:51.968671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:51.968684 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:51.968755 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:51.969778 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:51.970293 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:51.971895 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:51.972220 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:51.972307 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:51.974521 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:51.974522 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:51.974523 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:51.974531 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:51.976215 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:51.976618 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:51.976619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:51.976620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:51.976687 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:51.976929 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:51.976930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:51.976930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:51.977086 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:51.977167 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:51.977464 (+ 297us) server_negotiation.cc:299] Negotiation successful
0929 10:54:51.977564 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 10:54:51.977720 14588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:51.967846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:41611 (local address 127.10.238.194:55927)
0929 10:54:51.968009 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:51.968053 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:51.968559 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:51.968821 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:51.968825 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:51.969472 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:54:51.969657 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:51.969662 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:51.970718 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:51.970719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:51.971793 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:51.971795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:51.972323 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:51.972324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:51.972418 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:51.972977 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:54:51.972986 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:54:51.974427 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:54:51.976307 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:51.976314 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:51.976316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:51.976542 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:51.976767 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:51.976768 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:51.976768 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:51.976863 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:51.977157 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:54:51.977161 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:54:51.977326 (+ 165us) client_negotiation.cc:769] Sending connection context
0929 10:54:51.977413 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 10:54:51.977531 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":85,"threads_started":1}
I20260929 10:54:51.978195 14586 heartbeater.cc:347] Connected to a master server at 127.10.238.254:41611
I20260929 10:54:51.978312 14586 heartbeater.cc:464] Registering TS with master...
I20260929 10:54:51.978500 14586 heartbeater.cc:511] Master 127.10.238.254:41611 requested a full tablet report, sending...
I20260929 10:54:51.978909 14261 ts_manager.cc:194] Registered new tserver with Master: 75aae7dfcd4b4e4db94f7a674f1e4def (127.10.238.194:37155)
I20260929 10:54:51.979274 14261 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.194@KRBTEST.COM'} at 127.10.238.194:55927
WARNING: no policy specified for kudu/127.10.238.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab.
Entry for principal kudu/127.10.238.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab.
I20260929 10:54:51.996534 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.195:0
--local_ip_for_outbound_sockets=127.10.238.195
--webserver_interface=127.10.238.195
--webserver_port=0
--tserver_master_addrs=127.10.238.254:41611
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab
--principal=kudu/127.10.238.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:45027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:54:52.076485 14591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:54:52.076725 14591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:54:52.076764 14591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:54:52.077932 14591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:54:52.077999 14591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:54:52.078043 14591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.195
I20260929 10:54:52.079483 14591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:45027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab
--principal=kudu/127.10.238.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.238.195
--webserver_port=0
--tserver_master_addrs=127.10.238.254:41611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.195
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:54:52.079744 14591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:54:52.079977 14591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:54:52.084954 14599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:52.085016 14596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:52.085029 14597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:52.085179 14591 server_base.cc:1034] running on GCE node
I20260929 10:54:52.085462 14591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:54:52.085686 14591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:54:52.086822 14591 hybrid_clock.cc:648] HybridClock initialized: now 1790679292086812 us; error 28 us; skew 500 ppm
Sep 29 10:54:52 dist-test-slave-8nrt krb5kdc[14216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679292, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:54:52.089177 14591 init.cc:377] Logged in from keytab as kudu/127.10.238.195@KRBTEST.COM (short username kudu)
I20260929 10:54:52.089659 14591 webserver.cc:460] Webserver started at http://127.10.238.195:33921/ using document root <none> and password file <none>
I20260929 10:54:52.089874 14591 fs_manager.cc:359] Metadata directory not provided
I20260929 10:54:52.089932 14591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:54:52.090111 14591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:54:52.091111 14591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-2/data/instance:
uuid: "c823298512ed4c338b99431b1087154c"
format_stamp: "Formatted at 2026-09-29 10:54:52 on dist-test-slave-8nrt"
server_key: "2cc253f0a451d5848e043abb65cc4425"
server_key_iv: "7fd74d2bd0512992654b556c6e164bf1"
server_key_version: "encryptionkey@0"
I20260929 10:54:52.091478 14591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-2/wal/instance:
uuid: "c823298512ed4c338b99431b1087154c"
format_stamp: "Formatted at 2026-09-29 10:54:52 on dist-test-slave-8nrt"
server_key: "2cc253f0a451d5848e043abb65cc4425"
server_key_iv: "7fd74d2bd0512992654b556c6e164bf1"
server_key_version: "encryptionkey@0"
I20260929 10:54:52.092752 14591 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 10:54:52.093503 14606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:52.093683 14591 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 10:54:52.093761 14591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-2/wal
uuid: "c823298512ed4c338b99431b1087154c"
format_stamp: "Formatted at 2026-09-29 10:54:52 on dist-test-slave-8nrt"
server_key: "2cc253f0a451d5848e043abb65cc4425"
server_key_iv: "7fd74d2bd0512992654b556c6e164bf1"
server_key_version: "encryptionkey@0"
I20260929 10:54:52.093817 14591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:54:52.107107 14591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:54:52.109580 14591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:54:52.109691 14591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:54:52.109884 14591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:54:52.110136 14591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:54:52.110167 14591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:52.110188 14591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:54:52.110201 14591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:52.115970 14591 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.195:36617
I20260929 10:54:52.116053 14719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.195:36617 every 8 connection(s)
I20260929 10:54:52.116381 14591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
I20260929 10:54:52.121088 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 14591
I20260929 10:54:52.121169 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790679274510869-11195-0/minicluster-data/ts-2/wal/instance
I20260929 10:54:52.121423 11195 external_mini_cluster.cc:1428] Setting key 06e879da8e7bffaea42e10914fe66e0f
Sep 29 10:54:52 dist-test-slave-8nrt krb5kdc[14216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679292, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.195@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:54:52.125838 14300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:52.117184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:56383 (local address 127.10.238.254:41611)
0929 10:54:52.117291 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:52.117294 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:52.117458 (+ 164us) server_negotiation.cc:407] Connection header received
0929 10:54:52.118002 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:52.118003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:52.118015 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:52.118066 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:52.119034 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:52.119548 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:52.120613 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:52.120852 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:52.120965 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:52.123416 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:52.123418 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:52.123418 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:52.123425 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:52.124624 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:52.124984 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:52.124985 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:52.124985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:52.125035 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:52.125246 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:52.125247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:52.125247 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:52.125407 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:52.125460 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:52.125699 (+ 239us) server_negotiation.cc:299] Negotiation successful
0929 10:54:52.125774 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 10:54:52.126049 14722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:52.117157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:41611 (local address 127.10.238.195:56383)
0929 10:54:52.117339 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:52.117375 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:52.117900 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:52.118150 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:52.118154 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:52.118733 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:54:52.118927 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:52.118932 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:52.119649 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:52.119650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:52.120475 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:52.120477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:52.120979 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:52.120980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:52.121105 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:52.121731 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:54:52.121742 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:54:52.123332 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:54:52.124698 (+ 1366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:52.124704 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:52.124706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:52.124917 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:52.125094 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:52.125095 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:52.125095 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:52.125192 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:52.125465 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:54:52.125470 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:54:52.125588 (+ 118us) client_negotiation.cc:769] Sending connection context
0929 10:54:52.125673 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 10:54:52.125798 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":101,"threads_started":1}
I20260929 10:54:52.126612 14720 heartbeater.cc:347] Connected to a master server at 127.10.238.254:41611
I20260929 10:54:52.126720 14720 heartbeater.cc:464] Registering TS with master...
I20260929 10:54:52.126927 14720 heartbeater.cc:511] Master 127.10.238.254:41611 requested a full tablet report, sending...
I20260929 10:54:52.127287 14261 ts_manager.cc:194] Registered new tserver with Master: c823298512ed4c338b99431b1087154c (127.10.238.195:36617)
I20260929 10:54:52.127666 14261 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.195@KRBTEST.COM'} at 127.10.238.195:56383
I20260929 10:54:52.133291 11195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 10:54:52 dist-test-slave-8nrt krb5kdc[14216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679292, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 29 10:54:52 dist-test-slave-8nrt krb5kdc[14216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679292, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:54:52.156515 14300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:52.149154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43058 (local address 127.10.238.254:41611)
0929 10:54:52.149214 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:52.149216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:52.149361 (+ 145us) server_negotiation.cc:407] Connection header received
0929 10:54:52.149410 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:52.149410 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:52.149420 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:52.149478 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:52.150069 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:52.150530 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:52.151280 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:52.151460 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:52.151551 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:52.153991 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:52.153993 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:52.153994 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:52.154002 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:52.155438 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:52.155740 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:52.155741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:52.155741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:52.155799 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:52.155962 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:52.155963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:52.155963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:52.156118 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:52.156198 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:52.156384 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 10:54:52.156448 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260929 10:54:52.156790 14261 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43058
I20260929 10:54:52.157323 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 14323
I20260929 10:54:52.162161 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 14457
I20260929 10:54:52.166729 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 14591
I20260929 10:54:52.170993 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 14229
2026-09-29T10:54:52Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1816 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:54:52 dist-test-slave-8nrt krb5kdc[14732](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:54:52 dist-test-slave-8nrt krb5kdc[14732](info): set up 2 sockets
Sep 29 10:54:52 dist-test-slave-8nrt krb5kdc[14732](info): commencing operation
krb5kdc: starting...
W20260929 10:54:53.201076 11195 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Sep 29 10:54:53 dist-test-slave-8nrt krb5kdc[14732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:54:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:54:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
I20260929 10:54:53.320127 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:37181
--webserver_interface=127.10.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:46881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:37181
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:54:53.400118 14745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:54:53.400307 14745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:54:53.400328 14745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:54:53.401400 14745 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:54:53.401434 14745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:54:53.401448 14745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:54:53.401460 14745 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:54:53.401471 14745 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:54:53.402855 14745 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:46881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:37181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:37181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:54:53.403043 14745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:54:53.403208 14745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:54:53.408030 14751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:53.408046 14750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:53.408033 14753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:53.408311 14745 server_base.cc:1034] running on GCE node
I20260929 10:54:53.408567 14745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:54:53.408955 14745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:54:53.410136 14745 hybrid_clock.cc:648] HybridClock initialized: now 1790679293410119 us; error 45 us; skew 500 ppm
Sep 29 10:54:53 dist-test-slave-8nrt krb5kdc[14732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679293, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:54:53.412827 14745 init.cc:377] Logged in from keytab as kudu/127.10.238.254@KRBTEST.COM (short username kudu)
I20260929 10:54:53.413223 14745 webserver.cc:460] Webserver started at http://127.10.238.254:35543/ using document root <none> and password file <none>
I20260929 10:54:53.413471 14745 fs_manager.cc:359] Metadata directory not provided
I20260929 10:54:53.413519 14745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:54:53.413615 14745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:54:53.414644 14745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/master-0/data/instance:
uuid: "a312d9a33e1a4fdf885e8db964ccb65c"
format_stamp: "Formatted at 2026-09-29 10:54:53 on dist-test-slave-8nrt"
server_key: "7b4dcfd596216131d2704d89c78401b3"
server_key_iv: "87e83fb18160ff807fa2d3ca5b75d829"
server_key_version: "encryptionkey@0"
I20260929 10:54:53.414953 14745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/master-0/wal/instance:
uuid: "a312d9a33e1a4fdf885e8db964ccb65c"
format_stamp: "Formatted at 2026-09-29 10:54:53 on dist-test-slave-8nrt"
server_key: "7b4dcfd596216131d2704d89c78401b3"
server_key_iv: "87e83fb18160ff807fa2d3ca5b75d829"
server_key_version: "encryptionkey@0"
I20260929 10:54:53.416240 14745 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 10:54:53.416987 14760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:53.417162 14745 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 10:54:53.417248 14745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "a312d9a33e1a4fdf885e8db964ccb65c"
format_stamp: "Formatted at 2026-09-29 10:54:53 on dist-test-slave-8nrt"
server_key: "7b4dcfd596216131d2704d89c78401b3"
server_key_iv: "87e83fb18160ff807fa2d3ca5b75d829"
server_key_version: "encryptionkey@0"
I20260929 10:54:53.417356 14745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:54:53.435316 14745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:54:53.437337 14745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:54:53.437448 14745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:54:53.441293 14745 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.254:37181
I20260929 10:54:53.441368 14812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.254:37181 every 8 connection(s)
I20260929 10:54:53.441844 14745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:54:53.442544 14813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:54:53.444818 14813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a312d9a33e1a4fdf885e8db964ccb65c: Bootstrap starting.
I20260929 10:54:53.445108 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 14745
I20260929 10:54:53.445187 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/master-0/wal/instance
I20260929 10:54:53.445382 11195 external_mini_cluster.cc:1428] Setting key 5167e5ffbc0b4b1bf85a67a3edae2b99
I20260929 10:54:53.445562 14813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a312d9a33e1a4fdf885e8db964ccb65c: Neither blocks nor log segments found. Creating new log.
I20260929 10:54:53.445932 14813 log.cc:824] T 00000000000000000000000000000000 P a312d9a33e1a4fdf885e8db964ccb65c: Log is configured to *not* fsync() on all Append() calls
I20260929 10:54:53.446619 14813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a312d9a33e1a4fdf885e8db964ccb65c: No bootstrap required, opened a new log
I20260929 10:54:53.448913 14813 raft_consensus.cc:348] T 00000000000000000000000000000000 P a312d9a33e1a4fdf885e8db964ccb65c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a312d9a33e1a4fdf885e8db964ccb65c" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 37181 } }
I20260929 10:54:53.449080 14813 raft_consensus.cc:374] T 00000000000000000000000000000000 P a312d9a33e1a4fdf885e8db964ccb65c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:54:53.449112 14813 raft_consensus.cc:729] T 00000000000000000000000000000000 P a312d9a33e1a4fdf885e8db964ccb65c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a312d9a33e1a4fdf885e8db964ccb65c, State: Initialized, Role: FOLLOWER
I20260929 10:54:53.449221 14813 consensus_queue.cc:260] T 00000000000000000000000000000000 P a312d9a33e1a4fdf885e8db964ccb65c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a312d9a33e1a4fdf885e8db964ccb65c" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 37181 } }
I20260929 10:54:53.449319 14813 raft_consensus.cc:388] T 00000000000000000000000000000000 P a312d9a33e1a4fdf885e8db964ccb65c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:54:53.449360 14813 raft_consensus.cc:482] T 00000000000000000000000000000000 P a312d9a33e1a4fdf885e8db964ccb65c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:54:53.449409 14813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a312d9a33e1a4fdf885e8db964ccb65c [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:54:53.449981 14813 raft_consensus.cc:504] T 00000000000000000000000000000000 P a312d9a33e1a4fdf885e8db964ccb65c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a312d9a33e1a4fdf885e8db964ccb65c" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 37181 } }
I20260929 10:54:53.450089 14813 leader_election.cc:302] T 00000000000000000000000000000000 P a312d9a33e1a4fdf885e8db964ccb65c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a312d9a33e1a4fdf885e8db964ccb65c; no voters:
I20260929 10:54:53.450254 14813 leader_election.cc:288] T 00000000000000000000000000000000 P a312d9a33e1a4fdf885e8db964ccb65c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:54:53.450436 14819 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a312d9a33e1a4fdf885e8db964ccb65c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:54:53.450546 14813 sys_catalog.cc:564] T 00000000000000000000000000000000 P a312d9a33e1a4fdf885e8db964ccb65c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:54:53.450559 14819 raft_consensus.cc:686] T 00000000000000000000000000000000 P a312d9a33e1a4fdf885e8db964ccb65c [term 1 LEADER]: Becoming Leader. State: Replica: a312d9a33e1a4fdf885e8db964ccb65c, State: Running, Role: LEADER
I20260929 10:54:53.450729 14819 consensus_queue.cc:237] T 00000000000000000000000000000000 P a312d9a33e1a4fdf885e8db964ccb65c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a312d9a33e1a4fdf885e8db964ccb65c" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 37181 } }
I20260929 10:54:53.451076 14818 sys_catalog.cc:455] T 00000000000000000000000000000000 P a312d9a33e1a4fdf885e8db964ccb65c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a312d9a33e1a4fdf885e8db964ccb65c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a312d9a33e1a4fdf885e8db964ccb65c" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 37181 } } }
I20260929 10:54:53.451157 14818 sys_catalog.cc:458] T 00000000000000000000000000000000 P a312d9a33e1a4fdf885e8db964ccb65c [sys.catalog]: This master's current role is: LEADER
Sep 29 10:54:53 dist-test-slave-8nrt krb5kdc[14732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:54:53.452318 14820 sys_catalog.cc:455] T 00000000000000000000000000000000 P a312d9a33e1a4fdf885e8db964ccb65c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a312d9a33e1a4fdf885e8db964ccb65c. Latest consensus state: current_term: 1 leader_uuid: "a312d9a33e1a4fdf885e8db964ccb65c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a312d9a33e1a4fdf885e8db964ccb65c" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 37181 } } }
I20260929 10:54:53.452395 14820 sys_catalog.cc:458] T 00000000000000000000000000000000 P a312d9a33e1a4fdf885e8db964ccb65c [sys.catalog]: This master's current role is: LEADER
W20260929 10:54:53.452551 14833 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a312d9a33e1a4fdf885e8db964ccb65c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:54:53.452601 14833 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:54:53.452661 14826 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:54:53.452824 14826 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:54:53.454448 14826 catalog_manager.cc:1347] Generated new cluster ID: 9371274431414fc38e906c2d0042149f
I20260929 10:54:53.454500 14826 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:54:53.456728 14816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:53.445934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41114 (local address 127.10.238.254:37181)
0929 10:54:53.446139 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:53.446149 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:53.446184 (+ 35us) server_negotiation.cc:407] Connection header received
0929 10:54:53.446601 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:53.446607 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:53.446650 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:53.446807 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:53.447546 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:53.448361 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:53.449492 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:53.449913 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:53.449995 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:53.452866 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:53.452871 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:53.452871 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:53.452883 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:53.455046 (+ 2163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:53.455461 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:53.455464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:53.455465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:53.455576 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:53.455789 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:53.455790 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:53.455790 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:53.456098 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:53.456272 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:53.456370 (+ 98us) server_negotiation.cc:299] Negotiation successful
0929 10:54:53.456507 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":111,"threads_started":1}
I20260929 10:54:53.471540 14826 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:54:53.472224 14826 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:54:53.482319 14826 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a312d9a33e1a4fdf885e8db964ccb65c: Generated new TSK 0
I20260929 10:54:53.482528 14826 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.238.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab.
Entry for principal kudu/127.10.238.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab.
I20260929 10:54:53.512519 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.193:0
--local_ip_for_outbound_sockets=127.10.238.193
--webserver_interface=127.10.238.193
--webserver_port=0
--tserver_master_addrs=127.10.238.254:37181
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab
--principal=kudu/127.10.238.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:46881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:54:53.593227 14839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:54:53.593478 14839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:54:53.593516 14839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:54:53.594657 14839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:54:53.594717 14839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:54:53.594758 14839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.193
I20260929 10:54:53.596263 14839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:46881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab
--principal=kudu/127.10.238.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.238.193
--webserver_port=0
--tserver_master_addrs=127.10.238.254:37181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.193
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:54:53.596546 14839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:54:53.596767 14839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:54:53.601747 14844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:53.601794 14845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:53.601768 14847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:53.601998 14839 server_base.cc:1034] running on GCE node
I20260929 10:54:53.602286 14839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:54:53.602519 14839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:54:53.603672 14839 hybrid_clock.cc:648] HybridClock initialized: now 1790679293603659 us; error 29 us; skew 500 ppm
Sep 29 10:54:53 dist-test-slave-8nrt krb5kdc[14732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679293, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:54:53.606247 14839 init.cc:377] Logged in from keytab as kudu/127.10.238.193@KRBTEST.COM (short username kudu)
I20260929 10:54:53.606650 14839 webserver.cc:460] Webserver started at http://127.10.238.193:36203/ using document root <none> and password file <none>
I20260929 10:54:53.606891 14839 fs_manager.cc:359] Metadata directory not provided
I20260929 10:54:53.606956 14839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:54:53.607121 14839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:54:53.608187 14839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/ts-0/data/instance:
uuid: "dae02cd63d0d4005a41fe12b24914499"
format_stamp: "Formatted at 2026-09-29 10:54:53 on dist-test-slave-8nrt"
server_key: "6a600815abeb03a1f08cf53b049db921"
server_key_iv: "ba5dc69b7a2d8ce1f1dcaadd449c9814"
server_key_version: "encryptionkey@0"
I20260929 10:54:53.608547 14839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/ts-0/wal/instance:
uuid: "dae02cd63d0d4005a41fe12b24914499"
format_stamp: "Formatted at 2026-09-29 10:54:53 on dist-test-slave-8nrt"
server_key: "6a600815abeb03a1f08cf53b049db921"
server_key_iv: "ba5dc69b7a2d8ce1f1dcaadd449c9814"
server_key_version: "encryptionkey@0"
I20260929 10:54:53.609931 14839 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 10:54:53.610639 14854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:53.610821 14839 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:54:53.610924 14839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/ts-0/wal
uuid: "dae02cd63d0d4005a41fe12b24914499"
format_stamp: "Formatted at 2026-09-29 10:54:53 on dist-test-slave-8nrt"
server_key: "6a600815abeb03a1f08cf53b049db921"
server_key_iv: "ba5dc69b7a2d8ce1f1dcaadd449c9814"
server_key_version: "encryptionkey@0"
I20260929 10:54:53.610998 14839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:54:53.619050 14839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:54:53.620836 14839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:54:53.621016 14839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:54:53.621248 14839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:54:53.621595 14839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:54:53.621650 14839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:53.621682 14839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:54:53.621708 14839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:53.627292 14839 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.193:44399
I20260929 10:54:53.627377 14967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.193:44399 every 8 connection(s)
I20260929 10:54:53.627662 14839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
I20260929 10:54:53.628075 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 14839
I20260929 10:54:53.628206 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/ts-0/wal/instance
I20260929 10:54:53.628396 11195 external_mini_cluster.cc:1428] Setting key 404a223f81c1298bdaa6df112eb7930b
Sep 29 10:54:53 dist-test-slave-8nrt krb5kdc[14732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679293, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.193@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.238.194@KRBTEST.COM; defaulting to no policy
I20260929 10:54:53.637185 14816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:53.628428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:46211 (local address 127.10.238.254:37181)
0929 10:54:53.628477 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:53.628479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:53.628732 (+ 253us) server_negotiation.cc:407] Connection header received
0929 10:54:53.629852 (+ 1120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:53.629853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:53.629866 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:53.629938 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:53.630517 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:53.631030 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:53.632022 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:53.632251 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:53.632341 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:53.634486 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:53.634488 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:53.634488 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:53.634495 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:53.635899 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:53.636324 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:53.636326 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:53.636326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:53.636406 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:53.636646 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:53.636646 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:53.636647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:53.636764 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:53.636821 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:53.637050 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 10:54:53.637120 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 10:54:53.637356 14970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:53.628411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:37181 (local address 127.10.238.193:46211)
0929 10:54:53.628607 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:53.628653 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:53.629742 (+ 1089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:53.630001 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:53.630004 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:53.630233 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:54:53.630416 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:53.630419 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:53.631125 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:53.631132 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:53.631927 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:53.631929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:53.632353 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:53.632354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:53.632491 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:53.633005 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:54:53.633014 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:54:53.634384 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:54:53.635994 (+ 1610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:53.636002 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:53.636004 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:53.636240 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:53.636479 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:53.636480 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:53.636480 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:53.636581 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:53.636825 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:54:53.636829 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:54:53.636938 (+ 109us) client_negotiation.cc:769] Sending connection context
0929 10:54:53.637020 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 10:54:53.637130 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":100,"threads_started":1}
I20260929 10:54:53.637851 14968 heartbeater.cc:347] Connected to a master server at 127.10.238.254:37181
I20260929 10:54:53.637944 14968 heartbeater.cc:464] Registering TS with master...
I20260929 10:54:53.638077 14968 heartbeater.cc:511] Master 127.10.238.254:37181 requested a full tablet report, sending...
I20260929 10:54:53.638579 14777 ts_manager.cc:194] Registered new tserver with Master: dae02cd63d0d4005a41fe12b24914499 (127.10.238.193:44399)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.194@KRBTEST.COM" created.
I20260929 10:54:53.639230 14777 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.193@KRBTEST.COM'} at 127.10.238.193:46211
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab.
Entry for principal kudu/127.10.238.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab.
I20260929 10:54:53.653637 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.194:0
--local_ip_for_outbound_sockets=127.10.238.194
--webserver_interface=127.10.238.194
--webserver_port=0
--tserver_master_addrs=127.10.238.254:37181
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab
--principal=kudu/127.10.238.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:46881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:54:53.734269 14973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:54:53.734467 14973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:54:53.734488 14973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:54:53.735642 14973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:54:53.735684 14973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:54:53.735710 14973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.194
I20260929 10:54:53.737228 14973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:46881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab
--principal=kudu/127.10.238.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.238.194
--webserver_port=0
--tserver_master_addrs=127.10.238.254:37181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.194
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:54:53.737475 14973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:54:53.737705 14973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:54:53.742599 14978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:53.742599 14973 server_base.cc:1034] running on GCE node
W20260929 10:54:53.742573 14979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:53.742573 14981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:53.743059 14973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:54:53.743297 14973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:54:53.744436 14973 hybrid_clock.cc:648] HybridClock initialized: now 1790679293744417 us; error 33 us; skew 500 ppm
Sep 29 10:54:53 dist-test-slave-8nrt krb5kdc[14732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679293, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:54:53.746903 14973 init.cc:377] Logged in from keytab as kudu/127.10.238.194@KRBTEST.COM (short username kudu)
I20260929 10:54:53.747352 14973 webserver.cc:460] Webserver started at http://127.10.238.194:46763/ using document root <none> and password file <none>
I20260929 10:54:53.747563 14973 fs_manager.cc:359] Metadata directory not provided
I20260929 10:54:53.747627 14973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:54:53.747771 14973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:54:53.748768 14973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/ts-1/data/instance:
uuid: "ea0cd880a2844afba4d14ced8b44f973"
format_stamp: "Formatted at 2026-09-29 10:54:53 on dist-test-slave-8nrt"
server_key: "6edd2a2a9a8a3dbcc8dfdfa0d1a3c642"
server_key_iv: "e56c395fe3dbd8489a20a9ef8c415ef7"
server_key_version: "encryptionkey@0"
I20260929 10:54:53.749122 14973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/ts-1/wal/instance:
uuid: "ea0cd880a2844afba4d14ced8b44f973"
format_stamp: "Formatted at 2026-09-29 10:54:53 on dist-test-slave-8nrt"
server_key: "6edd2a2a9a8a3dbcc8dfdfa0d1a3c642"
server_key_iv: "e56c395fe3dbd8489a20a9ef8c415ef7"
server_key_version: "encryptionkey@0"
I20260929 10:54:53.750458 14973 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 10:54:53.751271 14988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:53.751515 14973 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 10:54:53.751614 14973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/ts-1/wal
uuid: "ea0cd880a2844afba4d14ced8b44f973"
format_stamp: "Formatted at 2026-09-29 10:54:53 on dist-test-slave-8nrt"
server_key: "6edd2a2a9a8a3dbcc8dfdfa0d1a3c642"
server_key_iv: "e56c395fe3dbd8489a20a9ef8c415ef7"
server_key_version: "encryptionkey@0"
I20260929 10:54:53.751684 14973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:54:53.772817 14973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:54:53.775141 14973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:54:53.775315 14973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:54:53.775543 14973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:54:53.775846 14973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:54:53.775898 14973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:53.775940 14973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:54:53.775967 14973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:53.781921 14973 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.194:45067
I20260929 10:54:53.781970 15101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.194:45067 every 8 connection(s)
I20260929 10:54:53.782301 14973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
Sep 29 10:54:53 dist-test-slave-8nrt krb5kdc[14732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679293, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.194@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:54:53.788388 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 14973
I20260929 10:54:53.788491 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/ts-1/wal/instance
I20260929 10:54:53.788683 11195 external_mini_cluster.cc:1428] Setting key 44f70000b0a01796e2f5f58afb89ec68
I20260929 10:54:53.791751 14816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:53.783012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:53857 (local address 127.10.238.254:37181)
0929 10:54:53.783091 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:53.783093 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:53.783245 (+ 152us) server_negotiation.cc:407] Connection header received
0929 10:54:53.784165 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:53.784166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:53.784177 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:53.784230 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:53.784868 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:53.785362 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:53.786306 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:53.786481 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:53.786531 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:53.788630 (+ 2099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:53.788631 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:53.788631 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:53.788636 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:53.790169 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:53.790700 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:53.790700 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:53.790701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:53.790773 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:53.791073 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:53.791074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:53.791074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:53.791220 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:53.791292 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:53.791537 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 10:54:53.791623 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 10:54:53.791770 15104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:53.782964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:37181 (local address 127.10.238.194:53857)
0929 10:54:53.783130 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:53.783165 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:53.784058 (+ 893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:53.784312 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:53.784316 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:53.784567 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:54:53.784752 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:53.784755 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:53.785433 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:53.785433 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:53.786232 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:53.786234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:53.786548 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:53.786549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:53.786643 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:53.787077 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:54:53.787085 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:54:53.788525 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:54:53.790263 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:53.790270 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:53.790272 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:53.790605 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:53.790841 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:53.790842 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:53.790843 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:53.790995 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:53.791291 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:54:53.791296 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:54:53.791408 (+ 112us) client_negotiation.cc:769] Sending connection context
0929 10:54:53.791491 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 10:54:53.791608 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":92,"threads_started":1}
I20260929 10:54:53.792266 15102 heartbeater.cc:347] Connected to a master server at 127.10.238.254:37181
I20260929 10:54:53.792367 15102 heartbeater.cc:464] Registering TS with master...
I20260929 10:54:53.792551 15102 heartbeater.cc:511] Master 127.10.238.254:37181 requested a full tablet report, sending...
I20260929 10:54:53.792910 14777 ts_manager.cc:194] Registered new tserver with Master: ea0cd880a2844afba4d14ced8b44f973 (127.10.238.194:45067)
I20260929 10:54:53.793351 14777 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.194@KRBTEST.COM'} at 127.10.238.194:53857
WARNING: no policy specified for kudu/127.10.238.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab.
Entry for principal kudu/127.10.238.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab.
I20260929 10:54:53.813660 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.195:0
--local_ip_for_outbound_sockets=127.10.238.195
--webserver_interface=127.10.238.195
--webserver_port=0
--tserver_master_addrs=127.10.238.254:37181
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab
--principal=kudu/127.10.238.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:46881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:54:53.894376 15107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:54:53.894575 15107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:54:53.894598 15107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:54:53.895761 15107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:54:53.895800 15107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:54:53.895852 15107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.195
I20260929 10:54:53.897353 15107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:46881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab
--principal=kudu/127.10.238.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.238.195
--webserver_port=0
--tserver_master_addrs=127.10.238.254:37181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.195
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:54:53.897581 15107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:54:53.897814 15107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:54:53.902777 15115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:53.902765 15113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:53.902762 15112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:53.903142 15107 server_base.cc:1034] running on GCE node
I20260929 10:54:53.903395 15107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:54:53.903652 15107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:54:53.904834 15107 hybrid_clock.cc:648] HybridClock initialized: now 1790679293904798 us; error 35 us; skew 500 ppm
Sep 29 10:54:53 dist-test-slave-8nrt krb5kdc[14732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679293, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:54:53.907612 15107 init.cc:377] Logged in from keytab as kudu/127.10.238.195@KRBTEST.COM (short username kudu)
I20260929 10:54:53.908092 15107 webserver.cc:460] Webserver started at http://127.10.238.195:41471/ using document root <none> and password file <none>
I20260929 10:54:53.908324 15107 fs_manager.cc:359] Metadata directory not provided
I20260929 10:54:53.908389 15107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:54:53.908519 15107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:54:53.909641 15107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/ts-2/data/instance:
uuid: "fdff3f29049745f5a734029408a6f841"
format_stamp: "Formatted at 2026-09-29 10:54:53 on dist-test-slave-8nrt"
server_key: "bd609701f1f0dd89b8f3f27ce0e0aade"
server_key_iv: "21072904594a524df90875aef5c82439"
server_key_version: "encryptionkey@0"
I20260929 10:54:53.910017 15107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/ts-2/wal/instance:
uuid: "fdff3f29049745f5a734029408a6f841"
format_stamp: "Formatted at 2026-09-29 10:54:53 on dist-test-slave-8nrt"
server_key: "bd609701f1f0dd89b8f3f27ce0e0aade"
server_key_iv: "21072904594a524df90875aef5c82439"
server_key_version: "encryptionkey@0"
I20260929 10:54:53.911278 15107 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260929 10:54:53.911967 15122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:53.912137 15107 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:53.912227 15107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/ts-2/wal
uuid: "fdff3f29049745f5a734029408a6f841"
format_stamp: "Formatted at 2026-09-29 10:54:53 on dist-test-slave-8nrt"
server_key: "bd609701f1f0dd89b8f3f27ce0e0aade"
server_key_iv: "21072904594a524df90875aef5c82439"
server_key_version: "encryptionkey@0"
I20260929 10:54:53.912279 15107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:54:53.927393 15107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:54:53.929536 15107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:54:53.929652 15107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:54:53.929847 15107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:54:53.930171 15107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:54:53.930231 15107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:53.930266 15107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:54:53.930311 15107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:53.935972 15107 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.195:34941
I20260929 10:54:53.936030 15235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.195:34941 every 8 connection(s)
I20260929 10:54:53.936411 15107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
I20260929 10:54:53.938354 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 15107
I20260929 10:54:53.938462 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790679274510869-11195-0/minicluster-data/ts-2/wal/instance
I20260929 10:54:53.938657 11195 external_mini_cluster.cc:1428] Setting key 974abd2bdbdaf7a392d9d856caca80f4
Sep 29 10:54:53 dist-test-slave-8nrt krb5kdc[14732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679293, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.195@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:54:53.946170 14816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:53.937182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:36375 (local address 127.10.238.254:37181)
0929 10:54:53.937251 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:53.937271 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:53.937482 (+ 211us) server_negotiation.cc:407] Connection header received
0929 10:54:53.938565 (+ 1083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:53.938566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:53.938576 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:53.938636 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:53.939574 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:53.940100 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:53.941133 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:53.941407 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:53.941490 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:53.943557 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:53.943558 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:53.943558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:53.943565 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:53.944998 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:53.945408 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:53.945409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:53.945410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:53.945460 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:53.945672 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:53.945672 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:53.945673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:53.945774 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:53.945823 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:53.946048 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 10:54:53.946107 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 10:54:53.946393 15238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:53.937162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:37181 (local address 127.10.238.195:36375)
0929 10:54:53.937343 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:53.937385 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:53.938448 (+ 1063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:53.938711 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:53.938715 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:53.939156 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:54:53.939445 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:53.939451 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:53.940199 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:53.940200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:53.941029 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:53.941031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:53.941502 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:53.941503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:53.941638 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:53.942101 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:54:53.942108 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:54:53.943471 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:54:53.945066 (+ 1595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:53.945071 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:53.945073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:53.945340 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:53.945517 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:53.945517 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:53.945518 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:53.945617 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:53.945826 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:54:53.945830 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:54:53.945944 (+ 114us) client_negotiation.cc:769] Sending connection context
0929 10:54:53.946024 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 10:54:53.946139 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":118,"threads_started":1}
I20260929 10:54:53.946905 15236 heartbeater.cc:347] Connected to a master server at 127.10.238.254:37181
I20260929 10:54:53.946987 15236 heartbeater.cc:464] Registering TS with master...
I20260929 10:54:53.947175 15236 heartbeater.cc:511] Master 127.10.238.254:37181 requested a full tablet report, sending...
I20260929 10:54:53.947551 14777 ts_manager.cc:194] Registered new tserver with Master: fdff3f29049745f5a734029408a6f841 (127.10.238.195:34941)
I20260929 10:54:53.947906 14777 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.195@KRBTEST.COM'} at 127.10.238.195:36375
I20260929 10:54:53.950438 11195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 10:54:53 dist-test-slave-8nrt krb5kdc[14732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 10:54:53 dist-test-slave-8nrt krb5kdc[14732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.238.193@KRBTEST.COM
I20260929 10:54:53.972741 15243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:53.965379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49718 (local address 127.10.238.193:44399)
0929 10:54:53.965544 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:53.965548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:53.965557 (+ 9us) server_negotiation.cc:407] Connection header received
0929 10:54:53.965581 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:53.965583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:53.965596 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:53.965657 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:53.966181 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:53.966869 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:53.967554 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:53.967853 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:53.967905 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:53.970134 (+ 2229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:53.970143 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:53.970143 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:53.970156 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:53.971521 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:53.971895 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:53.971903 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:53.971903 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:53.971980 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:53.972194 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:53.972195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:53.972196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:53.972370 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:53.972497 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:53.972589 (+ 92us) server_negotiation.cc:299] Negotiation successful
0929 10:54:53.972659 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":85,"threads_started":1}
Sep 29 10:54:53 dist-test-slave-8nrt krb5kdc[14732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:54:53.980058 14816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:53.974015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41126 (local address 127.10.238.254:37181)
0929 10:54:53.974067 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:53.974068 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:53.974104 (+ 36us) server_negotiation.cc:407] Connection header received
0929 10:54:53.974124 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:53.974125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:53.974133 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:53.974165 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:53.974610 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:53.975060 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:53.975689 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:53.975865 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:53.975912 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:53.978017 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:53.978018 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:53.978018 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:53.978023 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:53.979190 (+ 1167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:53.979475 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:53.979476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:53.979476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:53.979522 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:53.979662 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:53.979662 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:53.979663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:53.979783 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:53.979854 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:53.979934 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 10:54:53.979999 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260929 10:54:53.980307 14777 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41126
I20260929 10:54:53.980682 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 14839
I20260929 10:54:53.985934 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 14973
I20260929 10:54:53.990429 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 15107
I20260929 10:54:53.994624 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 14745
2026-09-29T10:54:53Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1823 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:54:54 dist-test-slave-8nrt krb5kdc[15248](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:54:54 dist-test-slave-8nrt krb5kdc[15248](info): set up 2 sockets
Sep 29 10:54:54 dist-test-slave-8nrt krb5kdc[15248](info): commencing operation
krb5kdc: starting...
W20260929 10:54:55.029199 11195 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Sep 29 10:54:55 dist-test-slave-8nrt krb5kdc[15248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:54:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:54:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
I20260929 10:54:55.147755 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:43063
--webserver_interface=127.10.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:33933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:43063
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:54:55.227342 15261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:54:55.227591 15261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:54:55.227629 15261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:54:55.228771 15261 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:54:55.228847 15261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:54:55.228875 15261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:54:55.228902 15261 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:54:55.228927 15261 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:54:55.230427 15261 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:33933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:43063
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:43063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:54:55.230700 15261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:54:55.230928 15261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:54:55.235720 15266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:55.235702 15269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:55.235687 15267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:55.236022 15261 server_base.cc:1034] running on GCE node
I20260929 10:54:55.236266 15261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:54:55.236529 15261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:54:55.237725 15261 hybrid_clock.cc:648] HybridClock initialized: now 1790679295237706 us; error 30 us; skew 500 ppm
Sep 29 10:54:55 dist-test-slave-8nrt krb5kdc[15248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679295, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:54:55.240374 15261 init.cc:377] Logged in from keytab as kudu/127.10.238.254@KRBTEST.COM (short username kudu)
I20260929 10:54:55.240695 15261 fs_manager.cc:359] Metadata directory not provided
I20260929 10:54:55.240744 15261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:54:55.240835 15261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:54:55.242023 15261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/master-0/data/instance:
uuid: "99db5b53c2e546aea4b9cfdaf3582b56"
format_stamp: "Formatted at 2026-09-29 10:54:55 on dist-test-slave-8nrt"
server_key: "874399d05582b734ba399f399c70fd87"
server_key_iv: "365259fa734764a5118ff2899c484e09"
server_key_version: "encryptionkey@0"
I20260929 10:54:55.242368 15261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/master-0/wal/instance:
uuid: "99db5b53c2e546aea4b9cfdaf3582b56"
format_stamp: "Formatted at 2026-09-29 10:54:55 on dist-test-slave-8nrt"
server_key: "874399d05582b734ba399f399c70fd87"
server_key_iv: "365259fa734764a5118ff2899c484e09"
server_key_version: "encryptionkey@0"
I20260929 10:54:55.243657 15261 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 10:54:55.244319 15275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:55.244496 15261 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:54:55.244563 15261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "99db5b53c2e546aea4b9cfdaf3582b56"
format_stamp: "Formatted at 2026-09-29 10:54:55 on dist-test-slave-8nrt"
server_key: "874399d05582b734ba399f399c70fd87"
server_key_iv: "365259fa734764a5118ff2899c484e09"
server_key_version: "encryptionkey@0"
I20260929 10:54:55.244628 15261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:54:55.252175 15261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:54:55.253938 15261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:54:55.254062 15261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:54:55.258222 15261 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.254:43063
I20260929 10:54:55.258242 15327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.254:43063 every 8 connection(s)
I20260929 10:54:55.258543 15261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:54:55.259472 15328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:54:55.261688 15328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99db5b53c2e546aea4b9cfdaf3582b56: Bootstrap starting.
I20260929 10:54:55.262212 15328 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99db5b53c2e546aea4b9cfdaf3582b56: Neither blocks nor log segments found. Creating new log.
I20260929 10:54:55.262418 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 15261
I20260929 10:54:55.262534 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/master-0/wal/instance
I20260929 10:54:55.262559 15328 log.cc:824] T 00000000000000000000000000000000 P 99db5b53c2e546aea4b9cfdaf3582b56: Log is configured to *not* fsync() on all Append() calls
I20260929 10:54:55.262751 11195 external_mini_cluster.cc:1428] Setting key ad69b3fa7fa89d1e9013b513b65ad7ad
I20260929 10:54:55.263314 15328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99db5b53c2e546aea4b9cfdaf3582b56: No bootstrap required, opened a new log
I20260929 10:54:55.265976 15328 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99db5b53c2e546aea4b9cfdaf3582b56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99db5b53c2e546aea4b9cfdaf3582b56" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 43063 } }
I20260929 10:54:55.266145 15328 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99db5b53c2e546aea4b9cfdaf3582b56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:54:55.266182 15328 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99db5b53c2e546aea4b9cfdaf3582b56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99db5b53c2e546aea4b9cfdaf3582b56, State: Initialized, Role: FOLLOWER
I20260929 10:54:55.266306 15328 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99db5b53c2e546aea4b9cfdaf3582b56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99db5b53c2e546aea4b9cfdaf3582b56" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 43063 } }
I20260929 10:54:55.266378 15328 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99db5b53c2e546aea4b9cfdaf3582b56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:54:55.266405 15328 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99db5b53c2e546aea4b9cfdaf3582b56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:54:55.266440 15328 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99db5b53c2e546aea4b9cfdaf3582b56 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:54:55.267064 15328 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99db5b53c2e546aea4b9cfdaf3582b56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99db5b53c2e546aea4b9cfdaf3582b56" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 43063 } }
I20260929 10:54:55.267189 15328 leader_election.cc:302] T 00000000000000000000000000000000 P 99db5b53c2e546aea4b9cfdaf3582b56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99db5b53c2e546aea4b9cfdaf3582b56; no voters:
I20260929 10:54:55.267380 15328 leader_election.cc:288] T 00000000000000000000000000000000 P 99db5b53c2e546aea4b9cfdaf3582b56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:54:55.267405 15333 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99db5b53c2e546aea4b9cfdaf3582b56 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:54:55.267524 15333 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99db5b53c2e546aea4b9cfdaf3582b56 [term 1 LEADER]: Becoming Leader. State: Replica: 99db5b53c2e546aea4b9cfdaf3582b56, State: Running, Role: LEADER
I20260929 10:54:55.267637 15328 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99db5b53c2e546aea4b9cfdaf3582b56 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:54:55.267957 15333 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99db5b53c2e546aea4b9cfdaf3582b56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99db5b53c2e546aea4b9cfdaf3582b56" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 43063 } }
I20260929 10:54:55.268338 15335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99db5b53c2e546aea4b9cfdaf3582b56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99db5b53c2e546aea4b9cfdaf3582b56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99db5b53c2e546aea4b9cfdaf3582b56" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 43063 } } }
I20260929 10:54:55.268373 15334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99db5b53c2e546aea4b9cfdaf3582b56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99db5b53c2e546aea4b9cfdaf3582b56. Latest consensus state: current_term: 1 leader_uuid: "99db5b53c2e546aea4b9cfdaf3582b56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99db5b53c2e546aea4b9cfdaf3582b56" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 43063 } } }
I20260929 10:54:55.268437 15335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99db5b53c2e546aea4b9cfdaf3582b56 [sys.catalog]: This master's current role is: LEADER
I20260929 10:54:55.268461 15334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99db5b53c2e546aea4b9cfdaf3582b56 [sys.catalog]: This master's current role is: LEADER
I20260929 10:54:55.268702 15341 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:54:55.269315 15341 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 29 10:54:55 dist-test-slave-8nrt krb5kdc[15248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:54:55.270977 15341 catalog_manager.cc:1347] Generated new cluster ID: 459cf0ba21204a289a9d2eceb5cfe84e
I20260929 10:54:55.271029 15341 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:54:55.273972 15332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:55.263234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55804 (local address 127.10.238.254:43063)
0929 10:54:55.263416 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:55.263425 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:55.263449 (+ 24us) server_negotiation.cc:407] Connection header received
0929 10:54:55.263877 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:55.263884 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:55.263929 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:55.264037 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:55.264804 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:55.265762 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:55.266802 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:55.267217 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:55.267341 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:55.270181 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:55.270185 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:55.270185 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:55.270198 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:55.272119 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:55.272470 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:55.272473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:55.272473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:55.272563 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:55.272740 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:55.272740 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:55.272741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:55.272999 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:55.273145 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:55.273635 (+ 490us) server_negotiation.cc:299] Negotiation successful
0929 10:54:55.273768 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":96,"threads_started":1}
I20260929 10:54:55.284588 15341 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:54:55.285138 15341 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:54:55.294307 15341 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99db5b53c2e546aea4b9cfdaf3582b56: Generated new TSK 0
I20260929 10:54:55.294512 15341 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.238.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab.
Entry for principal kudu/127.10.238.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab.
I20260929 10:54:55.327533 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.193:0
--local_ip_for_outbound_sockets=127.10.238.193
--webserver_interface=127.10.238.193
--webserver_port=0
--tserver_master_addrs=127.10.238.254:43063
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab
--principal=kudu/127.10.238.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:33933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:54:55.407928 15354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:54:55.408152 15354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:54:55.408191 15354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:54:55.409394 15354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:54:55.409459 15354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:54:55.409507 15354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.193
I20260929 10:54:55.411005 15354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:33933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab
--principal=kudu/127.10.238.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.238.193
--webserver_port=0
--tserver_master_addrs=127.10.238.254:43063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.193
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:54:55.411262 15354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:54:55.411486 15354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:54:55.416275 15359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:55.416275 15362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:55.416301 15360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:55.416612 15354 server_base.cc:1034] running on GCE node
I20260929 10:54:55.416823 15354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:54:55.417016 15354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:54:55.418147 15354 hybrid_clock.cc:648] HybridClock initialized: now 1790679295418138 us; error 25 us; skew 500 ppm
Sep 29 10:54:55 dist-test-slave-8nrt krb5kdc[15248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679295, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:54:55.420619 15354 init.cc:377] Logged in from keytab as kudu/127.10.238.193@KRBTEST.COM (short username kudu)
I20260929 10:54:55.420981 15354 fs_manager.cc:359] Metadata directory not provided
I20260929 10:54:55.421030 15354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:54:55.421175 15354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:54:55.422224 15354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/ts-0/data/instance:
uuid: "c4e63614d4134117b27e0c33f7a34e19"
format_stamp: "Formatted at 2026-09-29 10:54:55 on dist-test-slave-8nrt"
server_key: "402cdd0badec1db5298f68262d71ea89"
server_key_iv: "598b80466a1011899dce33b0ed03cdce"
server_key_version: "encryptionkey@0"
I20260929 10:54:55.422583 15354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/ts-0/wal/instance:
uuid: "c4e63614d4134117b27e0c33f7a34e19"
format_stamp: "Formatted at 2026-09-29 10:54:55 on dist-test-slave-8nrt"
server_key: "402cdd0badec1db5298f68262d71ea89"
server_key_iv: "598b80466a1011899dce33b0ed03cdce"
server_key_version: "encryptionkey@0"
I20260929 10:54:55.423874 15354 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 10:54:55.424561 15368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:55.424737 15354 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:54:55.424839 15354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/ts-0/wal
uuid: "c4e63614d4134117b27e0c33f7a34e19"
format_stamp: "Formatted at 2026-09-29 10:54:55 on dist-test-slave-8nrt"
server_key: "402cdd0badec1db5298f68262d71ea89"
server_key_iv: "598b80466a1011899dce33b0ed03cdce"
server_key_version: "encryptionkey@0"
I20260929 10:54:55.424924 15354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:54:55.433815 15354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:54:55.435527 15354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:54:55.435678 15354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:54:55.435891 15354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:54:55.436208 15354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:54:55.436262 15354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:55.436308 15354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:54:55.436336 15354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:55.442294 15354 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.193:43305
I20260929 10:54:55.442332 15481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.193:43305 every 8 connection(s)
I20260929 10:54:55.442634 15354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
Sep 29 10:54:55 dist-test-slave-8nrt krb5kdc[15248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679295, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.193@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:54:55.451366 15332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:55.443347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:60701 (local address 127.10.238.254:43063)
0929 10:54:55.443408 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:55.443410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:55.443621 (+ 211us) server_negotiation.cc:407] Connection header received
0929 10:54:55.444086 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:55.444087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:55.444096 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:55.444153 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:55.445047 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:55.445620 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:55.446568 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:55.446761 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:55.446811 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:55.448856 (+ 2045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:55.448857 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:55.448858 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:55.448864 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:55.450277 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:55.450613 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:55.450614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:55.450614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:55.450663 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:55.450867 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:55.450867 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:55.450868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:55.450967 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:55.451016 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:55.451236 (+ 220us) server_negotiation.cc:299] Negotiation successful
0929 10:54:55.451301 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 10:54:55.451573 15484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:55.443335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43063 (local address 127.10.238.193:60701)
0929 10:54:55.443494 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:55.443536 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:55.444007 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:55.444226 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:55.444229 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:55.444485 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:54:55.444970 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:55.444975 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:55.445699 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:55.445700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:55.446495 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:55.446497 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:55.446843 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:55.446844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:55.446937 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:55.447436 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:54:55.447443 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:54:55.448773 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:54:55.450345 (+ 1572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:55.450350 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:55.450352 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:55.450549 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:55.450718 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:55.450719 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:55.450719 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:55.450813 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:55.451020 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:54:55.451024 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:54:55.451130 (+ 106us) client_negotiation.cc:769] Sending connection context
0929 10:54:55.451210 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 10:54:55.451329 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":83,"threads_started":1}
I20260929 10:54:55.452080 15482 heartbeater.cc:347] Connected to a master server at 127.10.238.254:43063
I20260929 10:54:55.452178 15482 heartbeater.cc:464] Registering TS with master...
I20260929 10:54:55.452309 15482 heartbeater.cc:511] Master 127.10.238.254:43063 requested a full tablet report, sending...
I20260929 10:54:55.452629 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 15354
I20260929 10:54:55.452756 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/ts-0/wal/instance
I20260929 10:54:55.452896 15292 ts_manager.cc:194] Registered new tserver with Master: c4e63614d4134117b27e0c33f7a34e19 (127.10.238.193:43305)
I20260929 10:54:55.452944 11195 external_mini_cluster.cc:1428] Setting key 6a06f72187c6379f03a5420c075bc0a3
I20260929 10:54:55.453809 15292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.193@KRBTEST.COM'} at 127.10.238.193:60701
WARNING: no policy specified for kudu/127.10.238.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab.
Entry for principal kudu/127.10.238.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab.
I20260929 10:54:55.475993 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.194:0
--local_ip_for_outbound_sockets=127.10.238.194
--webserver_interface=127.10.238.194
--webserver_port=0
--tserver_master_addrs=127.10.238.254:43063
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab
--principal=kudu/127.10.238.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:33933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:54:55.555099 15487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:54:55.555292 15487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:54:55.555316 15487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:54:55.556479 15487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:54:55.556521 15487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:54:55.556548 15487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.194
I20260929 10:54:55.558058 15487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:33933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab
--principal=kudu/127.10.238.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.238.194
--webserver_port=0
--tserver_master_addrs=127.10.238.254:43063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.194
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:54:55.558303 15487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:54:55.558522 15487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:54:55.563467 15493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:55.563509 15495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:55.563474 15492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:55.563699 15487 server_base.cc:1034] running on GCE node
I20260929 10:54:55.563916 15487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:54:55.564175 15487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:54:55.565322 15487 hybrid_clock.cc:648] HybridClock initialized: now 1790679295565323 us; error 45 us; skew 500 ppm
Sep 29 10:54:55 dist-test-slave-8nrt krb5kdc[15248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679295, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:54:55.567689 15487 init.cc:377] Logged in from keytab as kudu/127.10.238.194@KRBTEST.COM (short username kudu)
I20260929 10:54:55.568027 15487 fs_manager.cc:359] Metadata directory not provided
I20260929 10:54:55.568104 15487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:54:55.568213 15487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:54:55.569378 15487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/ts-1/data/instance:
uuid: "a5cb7f6aa8da4e7fb92d0574a4673215"
format_stamp: "Formatted at 2026-09-29 10:54:55 on dist-test-slave-8nrt"
server_key: "8287bafa524e8dada9117e28dda70415"
server_key_iv: "8971cb55eb4d751af5fb71cda39ed01e"
server_key_version: "encryptionkey@0"
I20260929 10:54:55.569687 15487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/ts-1/wal/instance:
uuid: "a5cb7f6aa8da4e7fb92d0574a4673215"
format_stamp: "Formatted at 2026-09-29 10:54:55 on dist-test-slave-8nrt"
server_key: "8287bafa524e8dada9117e28dda70415"
server_key_iv: "8971cb55eb4d751af5fb71cda39ed01e"
server_key_version: "encryptionkey@0"
I20260929 10:54:55.570941 15487 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:54:55.571563 15501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:55.571735 15487 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:54:55.571796 15487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/ts-1/wal
uuid: "a5cb7f6aa8da4e7fb92d0574a4673215"
format_stamp: "Formatted at 2026-09-29 10:54:55 on dist-test-slave-8nrt"
server_key: "8287bafa524e8dada9117e28dda70415"
server_key_iv: "8971cb55eb4d751af5fb71cda39ed01e"
server_key_version: "encryptionkey@0"
I20260929 10:54:55.571844 15487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:54:55.579578 15487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:54:55.581380 15487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:54:55.581488 15487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:54:55.581662 15487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:54:55.581914 15487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:54:55.581943 15487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:55.581964 15487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:54:55.581975 15487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:55.587584 15487 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.194:35713
I20260929 10:54:55.587663 15614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.194:35713 every 8 connection(s)
I20260929 10:54:55.587855 15487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
I20260929 10:54:55.590305 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 15487
I20260929 10:54:55.590384 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/ts-1/wal/instance
I20260929 10:54:55.590518 11195 external_mini_cluster.cc:1428] Setting key a8ad90d07864a787833b5402f78d2e3f
Sep 29 10:54:55 dist-test-slave-8nrt krb5kdc[15248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679295, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.194@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:54:55.597785 15332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:55.588531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:35625 (local address 127.10.238.254:43063)
0929 10:54:55.588576 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:55.588578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:55.588769 (+ 191us) server_negotiation.cc:407] Connection header received
0929 10:54:55.589214 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:55.589215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:55.589227 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:55.589323 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:55.590665 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:55.591387 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:55.592370 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:55.592575 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:55.592665 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:55.594774 (+ 2109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:55.594776 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:55.594776 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:55.594785 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:55.596419 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:55.596804 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:55.596806 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:55.596806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:55.596873 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:55.597101 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:55.597103 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:55.597103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:55.597248 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:55.597343 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:55.597581 (+ 238us) server_negotiation.cc:299] Negotiation successful
0929 10:54:55.597667 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 10:54:55.597812 15617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:55.588499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43063 (local address 127.10.238.194:35625)
0929 10:54:55.588648 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:55.588681 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:55.589133 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:55.589390 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:55.589393 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:55.589633 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:54:55.590105 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:55.590109 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:55.591470 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:55.591471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:55.592266 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:55.592268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:55.592677 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:55.592678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:55.592810 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:55.593340 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:54:55.593348 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:54:55.594678 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:54:55.596518 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:55.596524 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:55.596526 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:55.596729 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:55.596943 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:55.596943 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:55.596944 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:55.597035 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:55.597335 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:54:55.597340 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:54:55.597448 (+ 108us) client_negotiation.cc:769] Sending connection context
0929 10:54:55.597530 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 10:54:55.597652 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":90,"threads_started":1}
I20260929 10:54:55.598363 15615 heartbeater.cc:347] Connected to a master server at 127.10.238.254:43063
I20260929 10:54:55.598469 15615 heartbeater.cc:464] Registering TS with master...
I20260929 10:54:55.598624 15615 heartbeater.cc:511] Master 127.10.238.254:43063 requested a full tablet report, sending...
I20260929 10:54:55.599030 15292 ts_manager.cc:194] Registered new tserver with Master: a5cb7f6aa8da4e7fb92d0574a4673215 (127.10.238.194:35713)
WARNING: no policy specified for kudu/127.10.238.195@KRBTEST.COM; defaulting to no policy
I20260929 10:54:55.599432 15292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.194@KRBTEST.COM'} at 127.10.238.194:35625
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab.
Entry for principal kudu/127.10.238.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab.
I20260929 10:54:55.615453 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.195:0
--local_ip_for_outbound_sockets=127.10.238.195
--webserver_interface=127.10.238.195
--webserver_port=0
--tserver_master_addrs=127.10.238.254:43063
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab
--principal=kudu/127.10.238.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:33933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:54:55.695420 15620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:54:55.695616 15620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:54:55.695638 15620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:54:55.696785 15620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:54:55.696818 15620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:54:55.696844 15620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.195
I20260929 10:54:55.698344 15620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:33933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab
--principal=kudu/127.10.238.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.238.195
--webserver_port=0
--tserver_master_addrs=127.10.238.254:43063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.195
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:54:55.698518 15620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:54:55.698697 15620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:54:55.703713 15625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:55.703758 15620 server_base.cc:1034] running on GCE node
W20260929 10:54:55.703768 15626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:55.703717 15628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:55.704171 15620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:54:55.704450 15620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:54:55.705608 15620 hybrid_clock.cc:648] HybridClock initialized: now 1790679295705595 us; error 43 us; skew 500 ppm
Sep 29 10:54:55 dist-test-slave-8nrt krb5kdc[15248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679295, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:54:55.707964 15620 init.cc:377] Logged in from keytab as kudu/127.10.238.195@KRBTEST.COM (short username kudu)
I20260929 10:54:55.708312 15620 fs_manager.cc:359] Metadata directory not provided
I20260929 10:54:55.708380 15620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:54:55.708515 15620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:54:55.709623 15620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/ts-2/data/instance:
uuid: "5871a5a4c0514e14959ff50bbbeff8d1"
format_stamp: "Formatted at 2026-09-29 10:54:55 on dist-test-slave-8nrt"
server_key: "ddcc1886e902da20fe847a02776b67d3"
server_key_iv: "f0f7decf8bef39f89f287c3fda092fb6"
server_key_version: "encryptionkey@0"
I20260929 10:54:55.709972 15620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/ts-2/wal/instance:
uuid: "5871a5a4c0514e14959ff50bbbeff8d1"
format_stamp: "Formatted at 2026-09-29 10:54:55 on dist-test-slave-8nrt"
server_key: "ddcc1886e902da20fe847a02776b67d3"
server_key_iv: "f0f7decf8bef39f89f287c3fda092fb6"
server_key_version: "encryptionkey@0"
I20260929 10:54:55.711208 15620 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 10:54:55.711874 15634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:55.712040 15620 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:54:55.712133 15620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/ts-2/wal
uuid: "5871a5a4c0514e14959ff50bbbeff8d1"
format_stamp: "Formatted at 2026-09-29 10:54:55 on dist-test-slave-8nrt"
server_key: "ddcc1886e902da20fe847a02776b67d3"
server_key_iv: "f0f7decf8bef39f89f287c3fda092fb6"
server_key_version: "encryptionkey@0"
I20260929 10:54:55.712211 15620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:54:55.721701 15620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:54:55.723286 15620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:54:55.723435 15620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:54:55.723641 15620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:54:55.723933 15620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:54:55.723985 15620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:55.724023 15620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:54:55.724052 15620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:55.729887 15620 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.195:36507
I20260929 10:54:55.729928 15747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.195:36507 every 8 connection(s)
I20260929 10:54:55.730167 15620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
Sep 29 10:54:55 dist-test-slave-8nrt krb5kdc[15248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679295, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.195@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:54:55.738670 15332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:55.730831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:47961 (local address 127.10.238.254:43063)
0929 10:54:55.730896 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:55.730898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:55.731059 (+ 161us) server_negotiation.cc:407] Connection header received
0929 10:54:55.731502 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:55.731503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:55.731513 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:55.731565 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:55.732419 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:55.732914 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:55.733862 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:55.734053 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:55.734105 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:55.736067 (+ 1962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:55.736068 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:55.736069 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:55.736075 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:55.737552 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:55.737894 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:55.737895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:55.737895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:55.737943 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:55.738171 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:55.738171 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:55.738172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:55.738266 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:55.738313 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:55.738524 (+ 211us) server_negotiation.cc:299] Negotiation successful
0929 10:54:55.738603 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 10:54:55.738832 15750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:55.730778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43063 (local address 127.10.238.195:47961)
0929 10:54:55.730941 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:55.730979 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:55.731426 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:55.731614 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:55.731617 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:55.731859 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:54:55.732334 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:55.732339 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:55.732994 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:55.732995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:55.733789 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:55.733792 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:55.734126 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:55.734127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:55.734221 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:55.734664 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:54:55.734672 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:54:55.735988 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:54:55.737630 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:55.737635 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:55.737637 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:55.737832 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:55.738032 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:55.738033 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:55.738033 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:55.738121 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:55.738319 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:54:55.738323 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:54:55.738423 (+ 100us) client_negotiation.cc:769] Sending connection context
0929 10:54:55.738498 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 10:54:55.738621 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":81,"threads_started":1}
I20260929 10:54:55.739346 15748 heartbeater.cc:347] Connected to a master server at 127.10.238.254:43063
I20260929 10:54:55.739419 15748 heartbeater.cc:464] Registering TS with master...
I20260929 10:54:55.739528 15748 heartbeater.cc:511] Master 127.10.238.254:43063 requested a full tablet report, sending...
I20260929 10:54:55.739867 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 15620
I20260929 10:54:55.739883 15292 ts_manager.cc:194] Registered new tserver with Master: 5871a5a4c0514e14959ff50bbbeff8d1 (127.10.238.195:36507)
I20260929 10:54:55.739974 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/ts-2/wal/instance
I20260929 10:54:55.740168 11195 external_mini_cluster.cc:1428] Setting key f7e632acc328f00ad4ae50285d414df9
I20260929 10:54:55.740372 15292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.195@KRBTEST.COM'} at 127.10.238.195:47961
I20260929 10:54:55.740993 11195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:54:55.747577 15332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:55.742301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42610 (local address 127.10.238.254:43063)
0929 10:54:55.742344 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:55.742346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:55.742483 (+ 137us) server_negotiation.cc:407] Connection header received
0929 10:54:55.742500 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:55.742501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:55.742508 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:55.742543 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:55.743068 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:55.743522 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:55.744174 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:55.744369 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:55.744414 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:55.745443 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:55.745444 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:55.745444 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:55.745449 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:55.746529 (+ 1080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:55.746873 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:55.746874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:55.746874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:55.746919 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:55.747109 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:55.747110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:55.747110 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:55.747234 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:55.747314 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:55.747449 (+ 135us) server_negotiation.cc:299] Negotiation successful
0929 10:54:55.747502 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 10:54:55.749202 15291 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:54:55.750873 15291 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:54:55.757180 15761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:55.753248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57450 (local address 127.10.238.193:43305)
0929 10:54:55.753445 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:55.753449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:55.753457 (+ 8us) server_negotiation.cc:407] Connection header received
0929 10:54:55.753499 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:55.753501 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:55.753517 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:55.753601 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:54:55.753917 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:55.754652 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:55.756231 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:55.756766 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:55.756859 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:55.756917 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:55.757049 (+ 132us) server_negotiation.cc:299] Negotiation successful
0929 10:54:55.757097 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":98,"threads_started":1}
I20260929 10:54:55.757175 15757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:55.753009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:43305 (local address 127.0.0.1:57450)
0929 10:54:55.753252 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:55.753261 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:55.753330 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:55.753805 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:55.753805 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:55.753807 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:54:55.753855 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:55.753856 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:55.754752 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:55.754753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:55.756139 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:55.756141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:55.756858 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:55.756859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:55.756946 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:55.756960 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 10:54:55.757026 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 10:54:55.757090 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":2,"thread_start_us":64,"threads_started":1}
I20260929 10:54:55.757472 15759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:55.753095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47564 (local address 127.10.238.194:35713)
0929 10:54:55.753765 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:55.753770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:55.753780 (+ 10us) server_negotiation.cc:407] Connection header received
0929 10:54:55.753818 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:55.753820 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:55.753837 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:55.753950 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:54:55.754059 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:55.755116 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:55.756295 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:55.756763 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:55.756868 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:55.756929 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:55.757349 (+ 420us) server_negotiation.cc:299] Negotiation successful
0929 10:54:55.757400 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":79,"threads_started":1}
I20260929 10:54:55.757181 15756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:55.752985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.194:35713 (local address 127.0.0.1:47564)
0929 10:54:55.753150 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:55.753189 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:55.753254 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:55.753946 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:55.753947 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:55.753948 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:54:55.753999 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:55.754000 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:55.755217 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:55.755218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:55.756220 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:55.756223 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:55.756871 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:55.756872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:55.756969 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:55.756974 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 10:54:55.757032 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 10:54:55.757098 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":21,"thread_start_us":67,"threads_started":1}
I20260929 10:54:55.758016 15758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:55.753009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.195:36507 (local address 127.0.0.1:47236)
0929 10:54:55.753383 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:55.753390 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:55.753422 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:55.753636 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:55.753639 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:55.753641 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:54:55.753706 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:55.753708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:55.754990 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:55.754990 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:55.756138 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:55.756141 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:55.757766 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:55.757767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:55.757848 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:55.757852 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 10:54:55.757899 (+ 47us) client_negotiation.cc:240] Negotiation successful
0929 10:54:55.757944 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":44,"thread_start_us":89,"threads_started":1}
I20260929 10:54:55.758222 15760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:55.753232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47236 (local address 127.10.238.195:36507)
0929 10:54:55.753439 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:55.753445 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:55.753474 (+ 29us) server_negotiation.cc:407] Connection header received
0929 10:54:55.753503 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:55.753505 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:55.753519 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:55.753600 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:54:55.753793 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:55.754896 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:55.756952 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:55.757619 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:55.757689 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:55.757733 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:55.758112 (+ 379us) server_negotiation.cc:299] Negotiation successful
0929 10:54:55.758152 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":114,"threads_started":1}
I20260929 10:54:55.758224 15549 tablet_service.cc:1467] Processing CreateTablet for tablet 142fc766be7c417e99c8f0bbc07d3e50 (DEFAULT_TABLE table=test-table [id=367391e42491489688ce9c23525f2fdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:54:55.758358 15416 tablet_service.cc:1467] Processing CreateTablet for tablet 142fc766be7c417e99c8f0bbc07d3e50 (DEFAULT_TABLE table=test-table [id=367391e42491489688ce9c23525f2fdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:54:55.758608 15549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 142fc766be7c417e99c8f0bbc07d3e50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:54:55.758652 15416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 142fc766be7c417e99c8f0bbc07d3e50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:54:55.761094 15762 tablet_bootstrap.cc:492] T 142fc766be7c417e99c8f0bbc07d3e50 P a5cb7f6aa8da4e7fb92d0574a4673215: Bootstrap starting.
I20260929 10:54:55.761145 15763 tablet_bootstrap.cc:492] T 142fc766be7c417e99c8f0bbc07d3e50 P c4e63614d4134117b27e0c33f7a34e19: Bootstrap starting.
I20260929 10:54:55.761883 15763 tablet_bootstrap.cc:654] T 142fc766be7c417e99c8f0bbc07d3e50 P c4e63614d4134117b27e0c33f7a34e19: Neither blocks nor log segments found. Creating new log.
I20260929 10:54:55.761911 15762 tablet_bootstrap.cc:654] T 142fc766be7c417e99c8f0bbc07d3e50 P a5cb7f6aa8da4e7fb92d0574a4673215: Neither blocks nor log segments found. Creating new log.
I20260929 10:54:55.762056 15682 tablet_service.cc:1467] Processing CreateTablet for tablet 142fc766be7c417e99c8f0bbc07d3e50 (DEFAULT_TABLE table=test-table [id=367391e42491489688ce9c23525f2fdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:54:55.762434 15682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 142fc766be7c417e99c8f0bbc07d3e50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:54:55.762458 15763 log.cc:824] T 142fc766be7c417e99c8f0bbc07d3e50 P c4e63614d4134117b27e0c33f7a34e19: Log is configured to *not* fsync() on all Append() calls
I20260929 10:54:55.762493 15762 log.cc:824] T 142fc766be7c417e99c8f0bbc07d3e50 P a5cb7f6aa8da4e7fb92d0574a4673215: Log is configured to *not* fsync() on all Append() calls
I20260929 10:54:55.763187 15763 tablet_bootstrap.cc:492] T 142fc766be7c417e99c8f0bbc07d3e50 P c4e63614d4134117b27e0c33f7a34e19: No bootstrap required, opened a new log
I20260929 10:54:55.763231 15762 tablet_bootstrap.cc:492] T 142fc766be7c417e99c8f0bbc07d3e50 P a5cb7f6aa8da4e7fb92d0574a4673215: No bootstrap required, opened a new log
I20260929 10:54:55.763278 15763 ts_tablet_manager.cc:1397] T 142fc766be7c417e99c8f0bbc07d3e50 P c4e63614d4134117b27e0c33f7a34e19: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 10:54:55.763286 15762 ts_tablet_manager.cc:1397] T 142fc766be7c417e99c8f0bbc07d3e50 P a5cb7f6aa8da4e7fb92d0574a4673215: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 10:54:55.764807 15763 raft_consensus.cc:348] T 142fc766be7c417e99c8f0bbc07d3e50 P c4e63614d4134117b27e0c33f7a34e19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5cb7f6aa8da4e7fb92d0574a4673215" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35713 } } peers { permanent_uuid: "c4e63614d4134117b27e0c33f7a34e19" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43305 } } peers { permanent_uuid: "5871a5a4c0514e14959ff50bbbeff8d1" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36507 } }
I20260929 10:54:55.764963 15763 raft_consensus.cc:374] T 142fc766be7c417e99c8f0bbc07d3e50 P c4e63614d4134117b27e0c33f7a34e19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:54:55.765005 15763 raft_consensus.cc:729] T 142fc766be7c417e99c8f0bbc07d3e50 P c4e63614d4134117b27e0c33f7a34e19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4e63614d4134117b27e0c33f7a34e19, State: Initialized, Role: FOLLOWER
I20260929 10:54:55.765127 15763 consensus_queue.cc:260] T 142fc766be7c417e99c8f0bbc07d3e50 P c4e63614d4134117b27e0c33f7a34e19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5cb7f6aa8da4e7fb92d0574a4673215" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35713 } } peers { permanent_uuid: "c4e63614d4134117b27e0c33f7a34e19" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43305 } } peers { permanent_uuid: "5871a5a4c0514e14959ff50bbbeff8d1" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36507 } }
I20260929 10:54:55.765304 15762 raft_consensus.cc:348] T 142fc766be7c417e99c8f0bbc07d3e50 P a5cb7f6aa8da4e7fb92d0574a4673215 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5cb7f6aa8da4e7fb92d0574a4673215" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35713 } } peers { permanent_uuid: "c4e63614d4134117b27e0c33f7a34e19" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43305 } } peers { permanent_uuid: "5871a5a4c0514e14959ff50bbbeff8d1" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36507 } }
I20260929 10:54:55.765445 15482 heartbeater.cc:503] Master 127.10.238.254:43063 was elected leader, sending a full tablet report...
I20260929 10:54:55.765494 15762 raft_consensus.cc:374] T 142fc766be7c417e99c8f0bbc07d3e50 P a5cb7f6aa8da4e7fb92d0574a4673215 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:54:55.765542 15762 raft_consensus.cc:729] T 142fc766be7c417e99c8f0bbc07d3e50 P a5cb7f6aa8da4e7fb92d0574a4673215 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5cb7f6aa8da4e7fb92d0574a4673215, State: Initialized, Role: FOLLOWER
I20260929 10:54:55.765653 15762 consensus_queue.cc:260] T 142fc766be7c417e99c8f0bbc07d3e50 P a5cb7f6aa8da4e7fb92d0574a4673215 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5cb7f6aa8da4e7fb92d0574a4673215" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35713 } } peers { permanent_uuid: "c4e63614d4134117b27e0c33f7a34e19" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43305 } } peers { permanent_uuid: "5871a5a4c0514e14959ff50bbbeff8d1" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36507 } }
I20260929 10:54:55.765712 15763 ts_tablet_manager.cc:1428] T 142fc766be7c417e99c8f0bbc07d3e50 P c4e63614d4134117b27e0c33f7a34e19: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 10:54:55.765902 15767 tablet_bootstrap.cc:492] T 142fc766be7c417e99c8f0bbc07d3e50 P 5871a5a4c0514e14959ff50bbbeff8d1: Bootstrap starting.
I20260929 10:54:55.765947 15615 heartbeater.cc:503] Master 127.10.238.254:43063 was elected leader, sending a full tablet report...
I20260929 10:54:55.765939 15762 ts_tablet_manager.cc:1428] T 142fc766be7c417e99c8f0bbc07d3e50 P a5cb7f6aa8da4e7fb92d0574a4673215: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 10:54:55.766522 15767 tablet_bootstrap.cc:654] T 142fc766be7c417e99c8f0bbc07d3e50 P 5871a5a4c0514e14959ff50bbbeff8d1: Neither blocks nor log segments found. Creating new log.
I20260929 10:54:55.767094 15767 log.cc:824] T 142fc766be7c417e99c8f0bbc07d3e50 P 5871a5a4c0514e14959ff50bbbeff8d1: Log is configured to *not* fsync() on all Append() calls
I20260929 10:54:55.767737 15767 tablet_bootstrap.cc:492] T 142fc766be7c417e99c8f0bbc07d3e50 P 5871a5a4c0514e14959ff50bbbeff8d1: No bootstrap required, opened a new log
I20260929 10:54:55.767803 15767 ts_tablet_manager.cc:1397] T 142fc766be7c417e99c8f0bbc07d3e50 P 5871a5a4c0514e14959ff50bbbeff8d1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 10:54:55.769322 15767 raft_consensus.cc:348] T 142fc766be7c417e99c8f0bbc07d3e50 P 5871a5a4c0514e14959ff50bbbeff8d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5cb7f6aa8da4e7fb92d0574a4673215" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35713 } } peers { permanent_uuid: "c4e63614d4134117b27e0c33f7a34e19" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43305 } } peers { permanent_uuid: "5871a5a4c0514e14959ff50bbbeff8d1" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36507 } }
I20260929 10:54:55.769436 15767 raft_consensus.cc:374] T 142fc766be7c417e99c8f0bbc07d3e50 P 5871a5a4c0514e14959ff50bbbeff8d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:54:55.769456 15767 raft_consensus.cc:729] T 142fc766be7c417e99c8f0bbc07d3e50 P 5871a5a4c0514e14959ff50bbbeff8d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5871a5a4c0514e14959ff50bbbeff8d1, State: Initialized, Role: FOLLOWER
I20260929 10:54:55.769550 15767 consensus_queue.cc:260] T 142fc766be7c417e99c8f0bbc07d3e50 P 5871a5a4c0514e14959ff50bbbeff8d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5cb7f6aa8da4e7fb92d0574a4673215" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35713 } } peers { permanent_uuid: "c4e63614d4134117b27e0c33f7a34e19" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43305 } } peers { permanent_uuid: "5871a5a4c0514e14959ff50bbbeff8d1" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36507 } }
I20260929 10:54:55.769814 15767 ts_tablet_manager.cc:1428] T 142fc766be7c417e99c8f0bbc07d3e50 P 5871a5a4c0514e14959ff50bbbeff8d1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 10:54:55.769855 15748 heartbeater.cc:503] Master 127.10.238.254:43063 was elected leader, sending a full tablet report...
W20260929 10:54:55.838315 15616 tablet.cc:2369] T 142fc766be7c417e99c8f0bbc07d3e50 P a5cb7f6aa8da4e7fb92d0574a4673215: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 10:54:55.943423 15483 tablet.cc:2369] T 142fc766be7c417e99c8f0bbc07d3e50 P c4e63614d4134117b27e0c33f7a34e19: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 10:54:55.980682 15749 tablet.cc:2369] T 142fc766be7c417e99c8f0bbc07d3e50 P 5871a5a4c0514e14959ff50bbbeff8d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:54:55.982798 15768 raft_consensus.cc:482] T 142fc766be7c417e99c8f0bbc07d3e50 P a5cb7f6aa8da4e7fb92d0574a4673215 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:54:55.982921 15768 raft_consensus.cc:504] T 142fc766be7c417e99c8f0bbc07d3e50 P a5cb7f6aa8da4e7fb92d0574a4673215 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5cb7f6aa8da4e7fb92d0574a4673215" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35713 } } peers { permanent_uuid: "c4e63614d4134117b27e0c33f7a34e19" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43305 } } peers { permanent_uuid: "5871a5a4c0514e14959ff50bbbeff8d1" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36507 } }
I20260929 10:54:55.983362 15768 leader_election.cc:288] T 142fc766be7c417e99c8f0bbc07d3e50 P a5cb7f6aa8da4e7fb92d0574a4673215 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4e63614d4134117b27e0c33f7a34e19 (127.10.238.193:43305), 5871a5a4c0514e14959ff50bbbeff8d1 (127.10.238.195:36507)
I20260929 10:54:55.986359 15761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:55.983618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:38947 (local address 127.10.238.193:43305)
0929 10:54:55.983662 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:55.983664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:55.983672 (+ 8us) server_negotiation.cc:407] Connection header received
0929 10:54:55.983701 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:55.983702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:55.983712 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:55.983777 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:54:55.983978 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:55.984507 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:55.985633 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:55.985996 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:55.986052 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:55.986059 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:55.986241 (+ 182us) server_negotiation.cc:299] Negotiation successful
0929 10:54:55.986287 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 10:54:55.986388 15617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:55.983510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:43305 (local address 127.10.238.194:38947)
0929 10:54:55.983569 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:55.983580 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:55.983622 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:55.983812 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:55.983814 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:55.983818 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:54:55.983897 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:55.983900 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:55.984582 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:55.984584 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:55.985540 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:55.985543 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:55.986075 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:55.986076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:55.986157 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:55.986160 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 10:54:55.986212 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 10:54:55.986269 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":57}
I20260929 10:54:55.986394 15760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:55.983608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:45425 (local address 127.10.238.195:36507)
0929 10:54:55.983685 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:55.983686 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:55.983801 (+ 115us) server_negotiation.cc:407] Connection header received
0929 10:54:55.983849 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:55.983850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:55.983860 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:55.983927 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:54:55.984106 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:55.984572 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:55.985634 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:55.986024 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:55.986120 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:55.986128 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:55.986291 (+ 163us) server_negotiation.cc:299] Negotiation successful
0929 10:54:55.986330 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 10:54:55.986413 15771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:55.983534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.195:36507 (local address 127.10.238.194:45425)
0929 10:54:55.983734 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:55.983742 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:55.983792 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:55.983961 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:55.983962 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:55.983964 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:54:55.984037 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:55.984039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:55.984658 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:55.984659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:55.985543 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:55.985545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:55.986135 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:55.986136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:55.986192 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:55.986215 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 10:54:55.986273 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 10:54:55.986318 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":1,"thread_start_us":94,"threads_started":1}
I20260929 10:54:55.986728 15436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "142fc766be7c417e99c8f0bbc07d3e50" candidate_uuid: "a5cb7f6aa8da4e7fb92d0574a4673215" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4e63614d4134117b27e0c33f7a34e19" is_pre_election: true
I20260929 10:54:55.986781 15702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "142fc766be7c417e99c8f0bbc07d3e50" candidate_uuid: "a5cb7f6aa8da4e7fb92d0574a4673215" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5871a5a4c0514e14959ff50bbbeff8d1" is_pre_election: true
I20260929 10:54:55.986913 15436 raft_consensus.cc:2454] T 142fc766be7c417e99c8f0bbc07d3e50 P c4e63614d4134117b27e0c33f7a34e19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5cb7f6aa8da4e7fb92d0574a4673215 in term 0.
I20260929 10:54:55.986963 15702 raft_consensus.cc:2454] T 142fc766be7c417e99c8f0bbc07d3e50 P 5871a5a4c0514e14959ff50bbbeff8d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5cb7f6aa8da4e7fb92d0574a4673215 in term 0.
I20260929 10:54:55.987187 15502 leader_election.cc:302] T 142fc766be7c417e99c8f0bbc07d3e50 P a5cb7f6aa8da4e7fb92d0574a4673215 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5cb7f6aa8da4e7fb92d0574a4673215, c4e63614d4134117b27e0c33f7a34e19; no voters:
I20260929 10:54:55.987320 15768 raft_consensus.cc:2781] T 142fc766be7c417e99c8f0bbc07d3e50 P a5cb7f6aa8da4e7fb92d0574a4673215 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:54:55.987375 15768 raft_consensus.cc:482] T 142fc766be7c417e99c8f0bbc07d3e50 P a5cb7f6aa8da4e7fb92d0574a4673215 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:54:55.987396 15768 raft_consensus.cc:3037] T 142fc766be7c417e99c8f0bbc07d3e50 P a5cb7f6aa8da4e7fb92d0574a4673215 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:54:55.988029 15768 raft_consensus.cc:504] T 142fc766be7c417e99c8f0bbc07d3e50 P a5cb7f6aa8da4e7fb92d0574a4673215 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5cb7f6aa8da4e7fb92d0574a4673215" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35713 } } peers { permanent_uuid: "c4e63614d4134117b27e0c33f7a34e19" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43305 } } peers { permanent_uuid: "5871a5a4c0514e14959ff50bbbeff8d1" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36507 } }
I20260929 10:54:55.988165 15768 leader_election.cc:288] T 142fc766be7c417e99c8f0bbc07d3e50 P a5cb7f6aa8da4e7fb92d0574a4673215 [CANDIDATE]: Term 1 election: Requested vote from peers c4e63614d4134117b27e0c33f7a34e19 (127.10.238.193:43305), 5871a5a4c0514e14959ff50bbbeff8d1 (127.10.238.195:36507)
I20260929 10:54:55.988309 15436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "142fc766be7c417e99c8f0bbc07d3e50" candidate_uuid: "a5cb7f6aa8da4e7fb92d0574a4673215" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4e63614d4134117b27e0c33f7a34e19"
I20260929 10:54:55.988323 15702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "142fc766be7c417e99c8f0bbc07d3e50" candidate_uuid: "a5cb7f6aa8da4e7fb92d0574a4673215" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5871a5a4c0514e14959ff50bbbeff8d1"
I20260929 10:54:55.988386 15702 raft_consensus.cc:3037] T 142fc766be7c417e99c8f0bbc07d3e50 P 5871a5a4c0514e14959ff50bbbeff8d1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:54:55.988387 15436 raft_consensus.cc:3037] T 142fc766be7c417e99c8f0bbc07d3e50 P c4e63614d4134117b27e0c33f7a34e19 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:54:55.989315 15436 raft_consensus.cc:2454] T 142fc766be7c417e99c8f0bbc07d3e50 P c4e63614d4134117b27e0c33f7a34e19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5cb7f6aa8da4e7fb92d0574a4673215 in term 1.
I20260929 10:54:55.989315 15702 raft_consensus.cc:2454] T 142fc766be7c417e99c8f0bbc07d3e50 P 5871a5a4c0514e14959ff50bbbeff8d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5cb7f6aa8da4e7fb92d0574a4673215 in term 1.
I20260929 10:54:55.989464 15502 leader_election.cc:302] T 142fc766be7c417e99c8f0bbc07d3e50 P a5cb7f6aa8da4e7fb92d0574a4673215 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5871a5a4c0514e14959ff50bbbeff8d1, a5cb7f6aa8da4e7fb92d0574a4673215, c4e63614d4134117b27e0c33f7a34e19; no voters:
I20260929 10:54:55.989568 15768 raft_consensus.cc:2781] T 142fc766be7c417e99c8f0bbc07d3e50 P a5cb7f6aa8da4e7fb92d0574a4673215 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:54:55.989753 15768 raft_consensus.cc:686] T 142fc766be7c417e99c8f0bbc07d3e50 P a5cb7f6aa8da4e7fb92d0574a4673215 [term 1 LEADER]: Becoming Leader. State: Replica: a5cb7f6aa8da4e7fb92d0574a4673215, State: Running, Role: LEADER
I20260929 10:54:55.989847 15768 consensus_queue.cc:237] T 142fc766be7c417e99c8f0bbc07d3e50 P a5cb7f6aa8da4e7fb92d0574a4673215 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5cb7f6aa8da4e7fb92d0574a4673215" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35713 } } peers { permanent_uuid: "c4e63614d4134117b27e0c33f7a34e19" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43305 } } peers { permanent_uuid: "5871a5a4c0514e14959ff50bbbeff8d1" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36507 } }
I20260929 10:54:55.990703 15291 catalog_manager.cc:5495] T 142fc766be7c417e99c8f0bbc07d3e50 P a5cb7f6aa8da4e7fb92d0574a4673215 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5cb7f6aa8da4e7fb92d0574a4673215 (127.10.238.194). New cstate: current_term: 1 leader_uuid: "a5cb7f6aa8da4e7fb92d0574a4673215" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5cb7f6aa8da4e7fb92d0574a4673215" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35713 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4e63614d4134117b27e0c33f7a34e19" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5871a5a4c0514e14959ff50bbbeff8d1" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36507 } health_report { overall_health: UNKNOWN } } }
I20260929 10:54:56.041416 15759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:56.039198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47578 (local address 127.10.238.194:35713)
0929 10:54:56.039244 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:56.039250 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:56.039258 (+ 8us) server_negotiation.cc:407] Connection header received
0929 10:54:56.039297 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:56.039298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:56.039316 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:56.039377 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:54:56.039512 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:56.039985 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:56.040684 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:56.040865 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:56.040915 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:56.041040 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:54:56.041137 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:54:56.041190 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:56.041312 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 10:54:56.041339 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 10:54:56.043308 15702 raft_consensus.cc:1261] T 142fc766be7c417e99c8f0bbc07d3e50 P 5871a5a4c0514e14959ff50bbbeff8d1 [term 1 FOLLOWER]: Refusing update from remote peer a5cb7f6aa8da4e7fb92d0574a4673215: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:54:56.043373 15436 raft_consensus.cc:1261] T 142fc766be7c417e99c8f0bbc07d3e50 P c4e63614d4134117b27e0c33f7a34e19 [term 1 FOLLOWER]: Refusing update from remote peer a5cb7f6aa8da4e7fb92d0574a4673215: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:54:56.043694 15772 consensus_queue.cc:1035] T 142fc766be7c417e99c8f0bbc07d3e50 P a5cb7f6aa8da4e7fb92d0574a4673215 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4e63614d4134117b27e0c33f7a34e19" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 43305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:54:56.043819 15768 consensus_queue.cc:1035] T 142fc766be7c417e99c8f0bbc07d3e50 P a5cb7f6aa8da4e7fb92d0574a4673215 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5871a5a4c0514e14959ff50bbbeff8d1" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:54:56.047536 15777 mvcc.cc:205] Tried to move back new op lower bound from 7334622396590518272 to 7334622396375011328. Current Snapshot: MvccSnapshot[applied={T|T < 7334622396590518272 or (T in {7334622396590518272})}]
I20260929 10:54:56.047873 15776 mvcc.cc:205] Tried to move back new op lower bound from 7334622396590518272 to 7334622396375011328. Current Snapshot: MvccSnapshot[applied={T|T < 7334622396590518272 or (T in {7334622396590518272})}]
I20260929 10:54:56.049059 15292 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:54:56.049165 15292 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:54:56.049963 15292 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 99db5b53c2e546aea4b9cfdaf3582b56: Sending DeleteTablet for 3 replicas of tablet 142fc766be7c417e99c8f0bbc07d3e50
I20260929 10:54:56.050299 15549 tablet_service.cc:1514] Processing DeleteTablet for tablet 142fc766be7c417e99c8f0bbc07d3e50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:54:56 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:47564
I20260929 10:54:56.050313 15682 tablet_service.cc:1514] Processing DeleteTablet for tablet 142fc766be7c417e99c8f0bbc07d3e50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:54:56 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:47236
I20260929 10:54:56.050570 15416 tablet_service.cc:1514] Processing DeleteTablet for tablet 142fc766be7c417e99c8f0bbc07d3e50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:54:56 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:57450
I20260929 10:54:56.050611 15784 tablet_replica.cc:330] T 142fc766be7c417e99c8f0bbc07d3e50 P a5cb7f6aa8da4e7fb92d0574a4673215: stopping tablet replica
I20260929 10:54:56.050652 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 15354
I20260929 10:54:56.050696 15784 raft_consensus.cc:2229] T 142fc766be7c417e99c8f0bbc07d3e50 P a5cb7f6aa8da4e7fb92d0574a4673215 [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:54:56.050783 15784 raft_consensus.cc:2258] T 142fc766be7c417e99c8f0bbc07d3e50 P a5cb7f6aa8da4e7fb92d0574a4673215 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:54:56.051252 15784 ts_tablet_manager.cc:1905] T 142fc766be7c417e99c8f0bbc07d3e50 P a5cb7f6aa8da4e7fb92d0574a4673215: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:54:56.053189 15784 ts_tablet_manager.cc:1918] T 142fc766be7c417e99c8f0bbc07d3e50 P a5cb7f6aa8da4e7fb92d0574a4673215: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:54:56.053352 15784 log.cc:1196] T 142fc766be7c417e99c8f0bbc07d3e50 P a5cb7f6aa8da4e7fb92d0574a4673215: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/ts-1/wal/wals/142fc766be7c417e99c8f0bbc07d3e50
I20260929 10:54:56.053697 15784 ts_tablet_manager.cc:1939] T 142fc766be7c417e99c8f0bbc07d3e50 P a5cb7f6aa8da4e7fb92d0574a4673215: Deleting consensus metadata
I20260929 10:54:56.054167 15279 catalog_manager.cc:4841] TS a5cb7f6aa8da4e7fb92d0574a4673215 (127.10.238.194:35713): tablet 142fc766be7c417e99c8f0bbc07d3e50 (table test-table [id=367391e42491489688ce9c23525f2fdf]) successfully deleted
I20260929 10:54:56.056874 15785 tablet_replica.cc:330] T 142fc766be7c417e99c8f0bbc07d3e50 P 5871a5a4c0514e14959ff50bbbeff8d1: stopping tablet replica
W20260929 10:54:56.056864 15276 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.238.193:43305 (error 108)
I20260929 10:54:56.056989 15785 raft_consensus.cc:2229] T 142fc766be7c417e99c8f0bbc07d3e50 P 5871a5a4c0514e14959ff50bbbeff8d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:54:56.057148 15785 raft_consensus.cc:2258] T 142fc766be7c417e99c8f0bbc07d3e50 P 5871a5a4c0514e14959ff50bbbeff8d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:54:56.057482 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 15487
I20260929 10:54:56.057622 15758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:56.057495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:43305 (local address 127.0.0.1:57466)
0929 10:54:56.057547 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:56.057585 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.193:43305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260929 10:54:56.058032 15276 catalog_manager.cc:4584] TS c4e63614d4134117b27e0c33f7a34e19 (127.10.238.193:43305): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 142fc766be7c417e99c8f0bbc07d3e50: Network error: Client connection negotiation failed: client connection to 127.10.238.193:43305: connect: Connection refused (error 111)
I20260929 10:54:56.059239 15785 ts_tablet_manager.cc:1905] T 142fc766be7c417e99c8f0bbc07d3e50 P 5871a5a4c0514e14959ff50bbbeff8d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:54:56.061209 15785 ts_tablet_manager.cc:1918] T 142fc766be7c417e99c8f0bbc07d3e50 P 5871a5a4c0514e14959ff50bbbeff8d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:54:56.061301 15785 log.cc:1196] T 142fc766be7c417e99c8f0bbc07d3e50 P 5871a5a4c0514e14959ff50bbbeff8d1: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790679274510869-11195-0/minicluster-data/ts-2/wal/wals/142fc766be7c417e99c8f0bbc07d3e50
I20260929 10:54:56.061569 15785 ts_tablet_manager.cc:1939] T 142fc766be7c417e99c8f0bbc07d3e50 P 5871a5a4c0514e14959ff50bbbeff8d1: Deleting consensus metadata
I20260929 10:54:56.062146 15277 catalog_manager.cc:4841] TS 5871a5a4c0514e14959ff50bbbeff8d1 (127.10.238.195:36507): tablet 142fc766be7c417e99c8f0bbc07d3e50 (table test-table [id=367391e42491489688ce9c23525f2fdf]) successfully deleted
I20260929 10:54:56.063422 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 15620
I20260929 10:54:56.068084 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 15261
2026-09-29T10:54:56Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2073 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T10:54:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:54:56Z Disabled control of system clock
I20260929 10:54:56.095474 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:46359
--webserver_interface=127.10.238.254
--webserver_port=0
--builtin_ntp_servers=127.10.238.212:44429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:46359
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 10:54:56.175025 15791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:54:56.175238 15791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:54:56.175261 15791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:54:56.176440 15791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:54:56.176482 15791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:54:56.176496 15791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:54:56.176508 15791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:54:56.176522 15791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:54:56.178117 15791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:44429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:46359
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:46359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:54:56.178378 15791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:54:56.178684 15791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:54:56.183573 15799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:56.183565 15796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:56.183768 15797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:56.183948 15791 server_base.cc:1034] running on GCE node
I20260929 10:54:56.184208 15791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:54:56.184501 15791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:54:56.185671 15791 hybrid_clock.cc:648] HybridClock initialized: now 1790679296185662 us; error 36 us; skew 500 ppm
I20260929 10:54:56.186885 15791 webserver.cc:460] Webserver started at http://127.10.238.254:35733/ using document root <none> and password file <none>
I20260929 10:54:56.187141 15791 fs_manager.cc:359] Metadata directory not provided
I20260929 10:54:56.187213 15791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:54:56.187374 15791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:54:56.188501 15791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/master-0/data/instance:
uuid: "8e9d9278896248e790f55bd1ec081f46"
format_stamp: "Formatted at 2026-09-29 10:54:56 on dist-test-slave-8nrt"
server_key: "bbb61a8045cb7023296953348a33d8c7"
server_key_iv: "2efa079f8f4416bc4c16996f4f25eb13"
server_key_version: "encryptionkey@0"
I20260929 10:54:56.188869 15791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/master-0/wal/instance:
uuid: "8e9d9278896248e790f55bd1ec081f46"
format_stamp: "Formatted at 2026-09-29 10:54:56 on dist-test-slave-8nrt"
server_key: "bbb61a8045cb7023296953348a33d8c7"
server_key_iv: "2efa079f8f4416bc4c16996f4f25eb13"
server_key_version: "encryptionkey@0"
I20260929 10:54:56.190269 15791 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 10:54:56.191030 15805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:56.191190 15791 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 10:54:56.191298 15791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "8e9d9278896248e790f55bd1ec081f46"
format_stamp: "Formatted at 2026-09-29 10:54:56 on dist-test-slave-8nrt"
server_key: "bbb61a8045cb7023296953348a33d8c7"
server_key_iv: "2efa079f8f4416bc4c16996f4f25eb13"
server_key_version: "encryptionkey@0"
I20260929 10:54:56.191390 15791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:54:56.194830 15791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:54:56.195106 15791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:54:56.195248 15791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:54:56.199151 15791 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.254:46359
I20260929 10:54:56.199186 15857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.254:46359 every 8 connection(s)
I20260929 10:54:56.199519 15791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:54:56.200138 15858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:54:56.202875 15858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e9d9278896248e790f55bd1ec081f46: Bootstrap starting.
I20260929 10:54:56.203454 15858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e9d9278896248e790f55bd1ec081f46: Neither blocks nor log segments found. Creating new log.
I20260929 10:54:56.203756 15858 log.cc:824] T 00000000000000000000000000000000 P 8e9d9278896248e790f55bd1ec081f46: Log is configured to *not* fsync() on all Append() calls
I20260929 10:54:56.204423 15858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e9d9278896248e790f55bd1ec081f46: No bootstrap required, opened a new log
I20260929 10:54:56.206017 15858 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e9d9278896248e790f55bd1ec081f46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e9d9278896248e790f55bd1ec081f46" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 46359 } }
I20260929 10:54:56.206173 15858 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e9d9278896248e790f55bd1ec081f46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:54:56.206216 15858 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e9d9278896248e790f55bd1ec081f46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e9d9278896248e790f55bd1ec081f46, State: Initialized, Role: FOLLOWER
I20260929 10:54:56.206403 15858 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e9d9278896248e790f55bd1ec081f46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e9d9278896248e790f55bd1ec081f46" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 46359 } }
I20260929 10:54:56.206504 15858 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e9d9278896248e790f55bd1ec081f46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:54:56.206548 15858 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e9d9278896248e790f55bd1ec081f46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:54:56.206600 15858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e9d9278896248e790f55bd1ec081f46 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:54:56.207118 15858 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e9d9278896248e790f55bd1ec081f46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e9d9278896248e790f55bd1ec081f46" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 46359 } }
I20260929 10:54:56.207234 15858 leader_election.cc:302] T 00000000000000000000000000000000 P 8e9d9278896248e790f55bd1ec081f46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e9d9278896248e790f55bd1ec081f46; no voters:
I20260929 10:54:56.207408 15858 leader_election.cc:288] T 00000000000000000000000000000000 P 8e9d9278896248e790f55bd1ec081f46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:54:56.207441 15861 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e9d9278896248e790f55bd1ec081f46 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:54:56.207609 15861 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e9d9278896248e790f55bd1ec081f46 [term 1 LEADER]: Becoming Leader. State: Replica: 8e9d9278896248e790f55bd1ec081f46, State: Running, Role: LEADER
I20260929 10:54:56.207653 15858 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e9d9278896248e790f55bd1ec081f46 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:54:56.207747 15861 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e9d9278896248e790f55bd1ec081f46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e9d9278896248e790f55bd1ec081f46" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 46359 } }
I20260929 10:54:56.208144 15863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e9d9278896248e790f55bd1ec081f46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e9d9278896248e790f55bd1ec081f46. Latest consensus state: current_term: 1 leader_uuid: "8e9d9278896248e790f55bd1ec081f46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e9d9278896248e790f55bd1ec081f46" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 46359 } } }
I20260929 10:54:56.208308 15863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e9d9278896248e790f55bd1ec081f46 [sys.catalog]: This master's current role is: LEADER
I20260929 10:54:56.208367 15862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e9d9278896248e790f55bd1ec081f46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e9d9278896248e790f55bd1ec081f46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e9d9278896248e790f55bd1ec081f46" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 46359 } } }
I20260929 10:54:56.208462 15862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e9d9278896248e790f55bd1ec081f46 [sys.catalog]: This master's current role is: LEADER
I20260929 10:54:56.208791 15870 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:54:56.209316 15870 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:54:56.209527 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 15791
I20260929 10:54:56.209616 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/master-0/wal/instance
I20260929 10:54:56.209801 11195 external_mini_cluster.cc:1428] Setting key 919c30aa6fe15a090343791ea019f2ed
I20260929 10:54:56.211020 15870 catalog_manager.cc:1347] Generated new cluster ID: 09b6284b445542a5b396842257410079
I20260929 10:54:56.211086 15870 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:54:56.215019 15880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:56.210354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35924 (local address 127.10.238.254:46359)
0929 10:54:56.210576 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:56.210585 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:56.213528 (+ 2943us) server_negotiation.cc:407] Connection header received
0929 10:54:56.213674 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:56.213682 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:56.213726 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:56.213882 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:56.214535 (+ 653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:56.214539 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:56.214539 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:54:56.214544 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:56.214590 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:54:56.214605 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:56.214675 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:56.214769 (+ 94us) server_negotiation.cc:299] Negotiation successful
0929 10:54:56.214829 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":84,"threads_started":1}
I20260929 10:54:56.244192 15870 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:54:56.244406 15870 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:54:56.246979 15870 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e9d9278896248e790f55bd1ec081f46: Generated new TSK 0
I20260929 10:54:56.247198 15870 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:54:56.257922 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.193:0
--local_ip_for_outbound_sockets=127.10.238.193
--webserver_interface=127.10.238.193
--webserver_port=0
--tserver_master_addrs=127.10.238.254:46359
--builtin_ntp_servers=127.10.238.212:44429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 10:54:56.340493 15882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:54:56.340709 15882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:54:56.340732 15882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:54:56.342062 15882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:54:56.342106 15882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:54:56.342133 15882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.193
I20260929 10:54:56.343502 15882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:44429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.238.193
--webserver_port=0
--tserver_master_addrs=127.10.238.254:46359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.193
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:54:56.343777 15882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:54:56.344224 15882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:54:56.349359 15887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:56.349452 15882 server_base.cc:1034] running on GCE node
W20260929 10:54:56.349351 15890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:56.349582 15888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:56.349855 15882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:54:56.350111 15882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:54:56.351270 15882 hybrid_clock.cc:648] HybridClock initialized: now 1790679296351248 us; error 36 us; skew 500 ppm
I20260929 10:54:56.352609 15882 webserver.cc:460] Webserver started at http://127.10.238.193:43055/ using document root <none> and password file <none>
I20260929 10:54:56.352851 15882 fs_manager.cc:359] Metadata directory not provided
I20260929 10:54:56.352917 15882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:54:56.353063 15882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:54:56.354190 15882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/ts-0/data/instance:
uuid: "60f9d84ce2604b2481a2622b74b69749"
format_stamp: "Formatted at 2026-09-29 10:54:56 on dist-test-slave-8nrt"
server_key: "1057add4b89392bac9ba0f85d0a46707"
server_key_iv: "aa37244e1898b59f0e6bc5993059a0c8"
server_key_version: "encryptionkey@0"
I20260929 10:54:56.354553 15882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/ts-0/wal/instance:
uuid: "60f9d84ce2604b2481a2622b74b69749"
format_stamp: "Formatted at 2026-09-29 10:54:56 on dist-test-slave-8nrt"
server_key: "1057add4b89392bac9ba0f85d0a46707"
server_key_iv: "aa37244e1898b59f0e6bc5993059a0c8"
server_key_version: "encryptionkey@0"
I20260929 10:54:56.355855 15882 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 10:54:56.356631 15896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:56.356820 15882 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:54:56.356917 15882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/ts-0/wal
uuid: "60f9d84ce2604b2481a2622b74b69749"
format_stamp: "Formatted at 2026-09-29 10:54:56 on dist-test-slave-8nrt"
server_key: "1057add4b89392bac9ba0f85d0a46707"
server_key_iv: "aa37244e1898b59f0e6bc5993059a0c8"
server_key_version: "encryptionkey@0"
I20260929 10:54:56.357005 15882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:54:56.360411 15882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:54:56.360657 15882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:54:56.360800 15882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:54:56.361037 15882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:54:56.361399 15882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:54:56.361454 15882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:56.361492 15882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:54:56.361519 15882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:56.367241 15882 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.193:34055
I20260929 10:54:56.367339 16009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.193:34055 every 8 connection(s)
I20260929 10:54:56.367628 15882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
I20260929 10:54:56.370628 15880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:56.368340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:36513 (local address 127.10.238.254:46359)
0929 10:54:56.368399 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:56.368401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:56.368629 (+ 228us) server_negotiation.cc:407] Connection header received
0929 10:54:56.369134 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:56.369135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:56.369145 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:56.369213 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:56.370395 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:56.370397 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:56.370397 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:54:56.370402 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:56.370421 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:54:56.370426 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:56.370490 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:56.370549 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 10:54:56.370564 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 10:54:56.370782 16012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:56.368319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:46359 (local address 127.10.238.193:36513)
0929 10:54:56.368495 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:56.368550 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:56.369008 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:56.369346 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:56.369351 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:56.369361 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:54:56.370242 (+ 881us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:54:56.370242 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:54:56.370268 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:54:56.370304 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:54:56.370488 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:54:56.370490 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:54:56.370492 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 10:54:56.370541 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 10:54:56.370586 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":101,"threads_started":1}
I20260929 10:54:56.371268 16010 heartbeater.cc:347] Connected to a master server at 127.10.238.254:46359
I20260929 10:54:56.371351 16010 heartbeater.cc:464] Registering TS with master...
I20260929 10:54:56.371533 16010 heartbeater.cc:511] Master 127.10.238.254:46359 requested a full tablet report, sending...
I20260929 10:54:56.372056 15822 ts_manager.cc:194] Registered new tserver with Master: 60f9d84ce2604b2481a2622b74b69749 (127.10.238.193:34055)
I20260929 10:54:56.372895 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 15882
I20260929 10:54:56.372994 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/ts-0/wal/instance
I20260929 10:54:56.373167 11195 external_mini_cluster.cc:1428] Setting key 3a7d87fe92b9b890e39025affa8e4d2d
I20260929 10:54:56.374054 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.194:0
--local_ip_for_outbound_sockets=127.10.238.194
--webserver_interface=127.10.238.194
--webserver_port=0
--tserver_master_addrs=127.10.238.254:46359
--builtin_ntp_servers=127.10.238.212:44429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 10:54:56.456669 16013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:54:56.456881 16013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:54:56.456905 16013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:54:56.458138 16013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:54:56.458180 16013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:54:56.458207 16013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.194
I20260929 10:54:56.459694 16013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:44429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.238.194
--webserver_port=0
--tserver_master_addrs=127.10.238.254:46359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.194
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:54:56.459936 16013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:54:56.460235 16013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:54:56.465323 16021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:56.465464 16013 server_base.cc:1034] running on GCE node
W20260929 10:54:56.465323 16019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:56.465365 16018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:56.465808 16013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:54:56.466049 16013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:54:56.467196 16013 hybrid_clock.cc:648] HybridClock initialized: now 1790679296467178 us; error 39 us; skew 500 ppm
I20260929 10:54:56.468597 16013 webserver.cc:460] Webserver started at http://127.10.238.194:46875/ using document root <none> and password file <none>
I20260929 10:54:56.468793 16013 fs_manager.cc:359] Metadata directory not provided
I20260929 10:54:56.468834 16013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:54:56.468927 16013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:54:56.470021 16013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/ts-1/data/instance:
uuid: "e533016cc7bf4df28e3548b9bcf335f7"
format_stamp: "Formatted at 2026-09-29 10:54:56 on dist-test-slave-8nrt"
server_key: "aad981e6882515a1a728f68d13bee4cd"
server_key_iv: "5479dd3290afba9b31578d539de144be"
server_key_version: "encryptionkey@0"
I20260929 10:54:56.470331 16013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/ts-1/wal/instance:
uuid: "e533016cc7bf4df28e3548b9bcf335f7"
format_stamp: "Formatted at 2026-09-29 10:54:56 on dist-test-slave-8nrt"
server_key: "aad981e6882515a1a728f68d13bee4cd"
server_key_iv: "5479dd3290afba9b31578d539de144be"
server_key_version: "encryptionkey@0"
I20260929 10:54:56.471580 16013 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 10:54:56.472241 16027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:56.472419 16013 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:54:56.472517 16013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/ts-1/wal
uuid: "e533016cc7bf4df28e3548b9bcf335f7"
format_stamp: "Formatted at 2026-09-29 10:54:56 on dist-test-slave-8nrt"
server_key: "aad981e6882515a1a728f68d13bee4cd"
server_key_iv: "5479dd3290afba9b31578d539de144be"
server_key_version: "encryptionkey@0"
I20260929 10:54:56.472595 16013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:54:56.475932 16013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:54:56.476181 16013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:54:56.476328 16013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:54:56.476555 16013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:54:56.476855 16013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:54:56.476908 16013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:56.476948 16013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:54:56.476975 16013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:56.482959 16013 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.194:43863
I20260929 10:54:56.483006 16140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.194:43863 every 8 connection(s)
I20260929 10:54:56.483310 16013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
I20260929 10:54:56.486189 15880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:56.484044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:46165 (local address 127.10.238.254:46359)
0929 10:54:56.484133 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:56.484136 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:56.484333 (+ 197us) server_negotiation.cc:407] Connection header received
0929 10:54:56.484762 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:56.484763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:56.484768 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:56.484825 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:56.485971 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:56.485973 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:56.485973 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:54:56.485977 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:56.485995 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:54:56.486000 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:56.486057 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:56.486118 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 10:54:56.486132 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 10:54:56.486342 16143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:56.483985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:46359 (local address 127.10.238.194:46165)
0929 10:54:56.484211 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:56.484245 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:56.484679 (+ 434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:56.484874 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:56.484877 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:56.484884 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:54:56.485829 (+ 945us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:54:56.485830 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:54:56.485858 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:54:56.485886 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:54:56.486056 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:54:56.486059 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:54:56.486061 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 10:54:56.486111 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 10:54:56.486155 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":124,"threads_started":1}
I20260929 10:54:56.486878 16141 heartbeater.cc:347] Connected to a master server at 127.10.238.254:46359
I20260929 10:54:56.486963 16141 heartbeater.cc:464] Registering TS with master...
I20260929 10:54:56.487138 16141 heartbeater.cc:511] Master 127.10.238.254:46359 requested a full tablet report, sending...
I20260929 10:54:56.487542 15822 ts_manager.cc:194] Registered new tserver with Master: e533016cc7bf4df28e3548b9bcf335f7 (127.10.238.194:43863)
I20260929 10:54:56.488480 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 16013
I20260929 10:54:56.488598 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/ts-1/wal/instance
I20260929 10:54:56.488795 11195 external_mini_cluster.cc:1428] Setting key 80f3abcca20f3f8b8d02dca73994cee7
I20260929 10:54:56.489862 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.195:0
--local_ip_for_outbound_sockets=127.10.238.195
--webserver_interface=127.10.238.195
--webserver_port=0
--tserver_master_addrs=127.10.238.254:46359
--builtin_ntp_servers=127.10.238.212:44429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 10:54:56.572706 16144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:54:56.572922 16144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:54:56.572959 16144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:54:56.574139 16144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:54:56.574204 16144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:54:56.574244 16144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.195
I20260929 10:54:56.575649 16144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:44429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.238.195
--webserver_port=0
--tserver_master_addrs=127.10.238.254:46359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.195
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:54:56.575845 16144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:54:56.576150 16144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:54:56.581597 16149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:56.581611 16144 server_base.cc:1034] running on GCE node
W20260929 10:54:56.581588 16150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:56.581588 16152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:56.582043 16144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:54:56.582314 16144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:54:56.583503 16144 hybrid_clock.cc:648] HybridClock initialized: now 1790679296583489 us; error 26 us; skew 500 ppm
I20260929 10:54:56.584975 16144 webserver.cc:460] Webserver started at http://127.10.238.195:44049/ using document root <none> and password file <none>
I20260929 10:54:56.585222 16144 fs_manager.cc:359] Metadata directory not provided
I20260929 10:54:56.585315 16144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:54:56.585493 16144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:54:56.586624 16144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/ts-2/data/instance:
uuid: "3e02e47de5f2497593ca8ebb2fabaec5"
format_stamp: "Formatted at 2026-09-29 10:54:56 on dist-test-slave-8nrt"
server_key: "a27b4a04295dc7a3b87d27e7e5d79018"
server_key_iv: "6f61163fe7dde984ecc39a65a2d7166c"
server_key_version: "encryptionkey@0"
I20260929 10:54:56.587054 16144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/ts-2/wal/instance:
uuid: "3e02e47de5f2497593ca8ebb2fabaec5"
format_stamp: "Formatted at 2026-09-29 10:54:56 on dist-test-slave-8nrt"
server_key: "a27b4a04295dc7a3b87d27e7e5d79018"
server_key_iv: "6f61163fe7dde984ecc39a65a2d7166c"
server_key_version: "encryptionkey@0"
I20260929 10:54:56.588498 16144 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 10:54:56.589246 16158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:56.589471 16144 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 10:54:56.589573 16144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/ts-2/wal
uuid: "3e02e47de5f2497593ca8ebb2fabaec5"
format_stamp: "Formatted at 2026-09-29 10:54:56 on dist-test-slave-8nrt"
server_key: "a27b4a04295dc7a3b87d27e7e5d79018"
server_key_iv: "6f61163fe7dde984ecc39a65a2d7166c"
server_key_version: "encryptionkey@0"
I20260929 10:54:56.589666 16144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:54:56.593045 16144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:54:56.593375 16144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:54:56.593549 16144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:54:56.593825 16144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:54:56.594146 16144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:54:56.594205 16144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:56.594249 16144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:54:56.594278 16144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:56.600361 16144 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.195:40819
I20260929 10:54:56.600431 16271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.195:40819 every 8 connection(s)
I20260929 10:54:56.600780 16144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
I20260929 10:54:56.603844 15880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:56.601564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:48317 (local address 127.10.238.254:46359)
0929 10:54:56.601608 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:56.601610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:56.601904 (+ 294us) server_negotiation.cc:407] Connection header received
0929 10:54:56.602362 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:56.602363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:56.602369 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:56.602435 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:56.603611 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:56.603613 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:56.603613 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:54:56.603618 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:56.603637 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:54:56.603642 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:56.603706 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:56.603775 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 10:54:56.603789 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 10:54:56.603995 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:56.601544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:46359 (local address 127.10.238.195:48317)
0929 10:54:56.601724 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:56.601788 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:56.602256 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:56.602486 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:56.602490 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:56.602498 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:54:56.603453 (+ 955us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:54:56.603453 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:54:56.603481 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:54:56.603508 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:54:56.603705 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:54:56.603707 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:54:56.603710 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 10:54:56.603759 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 10:54:56.603811 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":100,"threads_started":1}
I20260929 10:54:56.604466 16272 heartbeater.cc:347] Connected to a master server at 127.10.238.254:46359
I20260929 10:54:56.604565 16272 heartbeater.cc:464] Registering TS with master...
I20260929 10:54:56.604722 16272 heartbeater.cc:511] Master 127.10.238.254:46359 requested a full tablet report, sending...
I20260929 10:54:56.604730 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 16144
I20260929 10:54:56.604831 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/ts-2/wal/instance
I20260929 10:54:56.605020 11195 external_mini_cluster.cc:1428] Setting key 8851602e0377ed8992570dcdcffdba32
I20260929 10:54:56.605100 15822 ts_manager.cc:194] Registered new tserver with Master: 3e02e47de5f2497593ca8ebb2fabaec5 (127.10.238.195:40819)
I20260929 10:54:56.605984 11195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:54:56.608485 15880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:56.607297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35940 (local address 127.10.238.254:46359)
0929 10:54:56.607336 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:56.607338 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:56.607553 (+ 215us) server_negotiation.cc:407] Connection header received
0929 10:54:56.607697 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:56.607698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:56.607704 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:56.607774 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:56.608309 (+ 535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:56.608310 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:56.608310 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:54:56.608314 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:56.608330 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:54:56.608333 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:56.608387 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:56.608426 (+ 39us) server_negotiation.cc:299] Negotiation successful
0929 10:54:56.608438 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 10:54:56.609977 15822 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35940:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:54:56.611482 15822 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:54:56.615579 16281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:56.614020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.194:43863 (local address 127.0.0.1:46746)
0929 10:54:56.614537 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:56.614546 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:56.614593 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:56.614777 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:56.614778 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:56.614781 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:54:56.615141 (+ 360us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:54:56.615141 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:54:56.615153 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:54:56.615169 (+ 16us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:54:56.615427 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:54:56.615432 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:54:56.615435 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 10:54:56.615500 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 10:54:56.615514 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":67,"threads_started":1}
I20260929 10:54:56.615684 16282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:56.613789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.195:40819 (local address 127.0.0.1:39248)
0929 10:54:56.614132 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:56.614142 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:56.614193 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:56.614429 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:56.614451 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:56.614457 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:54:56.615329 (+ 872us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:54:56.615330 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:54:56.615336 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:54:56.615367 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:54:56.615589 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:54:56.615590 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:54:56.615590 (+ 0us) client_negotiation.cc:769] Sending connection context
0929 10:54:56.615618 (+ 28us) client_negotiation.cc:240] Negotiation successful
0929 10:54:56.615627 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":2,"spinlock_wait_cycles":896,"thread_start_us":84,"threads_started":1}
I20260929 10:54:56.615682 16280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:56.613789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:34055 (local address 127.0.0.1:40184)
0929 10:54:56.613906 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:56.613948 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:56.614005 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:56.614968 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:56.614968 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:56.614970 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:54:56.615301 (+ 331us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:54:56.615302 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:54:56.615307 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:54:56.615315 (+ 8us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:54:56.615562 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:54:56.615563 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:54:56.615564 (+ 1us) client_negotiation.cc:769] Sending connection context
0929 10:54:56.615594 (+ 30us) client_negotiation.cc:240] Negotiation successful
0929 10:54:56.615607 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":66,"threads_started":1}
I20260929 10:54:56.615878 16285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:56.614341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40184 (local address 127.10.238.193:34055)
0929 10:54:56.614830 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:56.614835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:56.614845 (+ 10us) server_negotiation.cc:407] Connection header received
0929 10:54:56.614869 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:56.614872 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:56.614883 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:56.614957 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:56.615401 (+ 444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:56.615410 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:56.615410 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:54:56.615415 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:56.615450 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:54:56.615481 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:56.615725 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:56.615748 (+ 23us) server_negotiation.cc:299] Negotiation successful
0929 10:54:56.615764 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":92,"threads_started":1}
I20260929 10:54:56.615605 16283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:56.614130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46746 (local address 127.10.238.194:43863)
0929 10:54:56.614302 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:56.614306 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:56.614658 (+ 352us) server_negotiation.cc:407] Connection header received
0929 10:54:56.614693 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:56.614695 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:56.614707 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:56.614778 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:56.615275 (+ 497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:56.615287 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:56.615288 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:54:56.615293 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:56.615331 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:54:56.615370 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:56.615434 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:56.615510 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 10:54:56.615525 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":105,"threads_started":1}
I20260929 10:54:56.616042 16284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:56.614147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39248 (local address 127.10.238.195:40819)
0929 10:54:56.614298 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:56.614302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:56.614312 (+ 10us) server_negotiation.cc:407] Connection header received
0929 10:54:56.614335 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:56.614337 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:56.614345 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:56.614447 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:56.615456 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:56.615463 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:56.615464 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:54:56.615468 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:56.615497 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:54:56.615517 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:56.615916 (+ 399us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:56.615944 (+ 28us) server_negotiation.cc:299] Negotiation successful
0929 10:54:56.615961 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
I20260929 10:54:56.616735 16075 tablet_service.cc:1467] Processing CreateTablet for tablet 39ab9790d8db47e99f5ce2025ec30899 (DEFAULT_TABLE table=test-table [id=a08253cbcfa74c6ca6261b5848fcd7e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:54:56.616742 15944 tablet_service.cc:1467] Processing CreateTablet for tablet 39ab9790d8db47e99f5ce2025ec30899 (DEFAULT_TABLE table=test-table [id=a08253cbcfa74c6ca6261b5848fcd7e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:54:56.616747 16206 tablet_service.cc:1467] Processing CreateTablet for tablet 39ab9790d8db47e99f5ce2025ec30899 (DEFAULT_TABLE table=test-table [id=a08253cbcfa74c6ca6261b5848fcd7e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:54:56.617064 16075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39ab9790d8db47e99f5ce2025ec30899. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:54:56.617071 16206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39ab9790d8db47e99f5ce2025ec30899. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:54:56.617071 15944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39ab9790d8db47e99f5ce2025ec30899. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:54:56.619714 16286 tablet_bootstrap.cc:492] T 39ab9790d8db47e99f5ce2025ec30899 P e533016cc7bf4df28e3548b9bcf335f7: Bootstrap starting.
I20260929 10:54:56.620321 16287 tablet_bootstrap.cc:492] T 39ab9790d8db47e99f5ce2025ec30899 P 3e02e47de5f2497593ca8ebb2fabaec5: Bootstrap starting.
I20260929 10:54:56.620352 16286 tablet_bootstrap.cc:654] T 39ab9790d8db47e99f5ce2025ec30899 P e533016cc7bf4df28e3548b9bcf335f7: Neither blocks nor log segments found. Creating new log.
I20260929 10:54:56.620672 16286 log.cc:824] T 39ab9790d8db47e99f5ce2025ec30899 P e533016cc7bf4df28e3548b9bcf335f7: Log is configured to *not* fsync() on all Append() calls
I20260929 10:54:56.620781 16288 tablet_bootstrap.cc:492] T 39ab9790d8db47e99f5ce2025ec30899 P 60f9d84ce2604b2481a2622b74b69749: Bootstrap starting.
I20260929 10:54:56.620976 16287 tablet_bootstrap.cc:654] T 39ab9790d8db47e99f5ce2025ec30899 P 3e02e47de5f2497593ca8ebb2fabaec5: Neither blocks nor log segments found. Creating new log.
I20260929 10:54:56.621359 16287 log.cc:824] T 39ab9790d8db47e99f5ce2025ec30899 P 3e02e47de5f2497593ca8ebb2fabaec5: Log is configured to *not* fsync() on all Append() calls
I20260929 10:54:56.621483 16288 tablet_bootstrap.cc:654] T 39ab9790d8db47e99f5ce2025ec30899 P 60f9d84ce2604b2481a2622b74b69749: Neither blocks nor log segments found. Creating new log.
I20260929 10:54:56.621531 16286 tablet_bootstrap.cc:492] T 39ab9790d8db47e99f5ce2025ec30899 P e533016cc7bf4df28e3548b9bcf335f7: No bootstrap required, opened a new log
I20260929 10:54:56.621593 16286 ts_tablet_manager.cc:1397] T 39ab9790d8db47e99f5ce2025ec30899 P e533016cc7bf4df28e3548b9bcf335f7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 10:54:56.621816 16288 log.cc:824] T 39ab9790d8db47e99f5ce2025ec30899 P 60f9d84ce2604b2481a2622b74b69749: Log is configured to *not* fsync() on all Append() calls
I20260929 10:54:56.622329 16287 tablet_bootstrap.cc:492] T 39ab9790d8db47e99f5ce2025ec30899 P 3e02e47de5f2497593ca8ebb2fabaec5: No bootstrap required, opened a new log
I20260929 10:54:56.622411 16287 ts_tablet_manager.cc:1397] T 39ab9790d8db47e99f5ce2025ec30899 P 3e02e47de5f2497593ca8ebb2fabaec5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 10:54:56.622575 16288 tablet_bootstrap.cc:492] T 39ab9790d8db47e99f5ce2025ec30899 P 60f9d84ce2604b2481a2622b74b69749: No bootstrap required, opened a new log
I20260929 10:54:56.622646 16288 ts_tablet_manager.cc:1397] T 39ab9790d8db47e99f5ce2025ec30899 P 60f9d84ce2604b2481a2622b74b69749: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 10:54:56.624259 16287 raft_consensus.cc:348] T 39ab9790d8db47e99f5ce2025ec30899 P 3e02e47de5f2497593ca8ebb2fabaec5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f9d84ce2604b2481a2622b74b69749" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34055 } } peers { permanent_uuid: "3e02e47de5f2497593ca8ebb2fabaec5" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 40819 } } peers { permanent_uuid: "e533016cc7bf4df28e3548b9bcf335f7" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 43863 } }
I20260929 10:54:56.624286 16286 raft_consensus.cc:348] T 39ab9790d8db47e99f5ce2025ec30899 P e533016cc7bf4df28e3548b9bcf335f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f9d84ce2604b2481a2622b74b69749" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34055 } } peers { permanent_uuid: "3e02e47de5f2497593ca8ebb2fabaec5" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 40819 } } peers { permanent_uuid: "e533016cc7bf4df28e3548b9bcf335f7" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 43863 } }
I20260929 10:54:56.624416 16287 raft_consensus.cc:374] T 39ab9790d8db47e99f5ce2025ec30899 P 3e02e47de5f2497593ca8ebb2fabaec5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:54:56.624441 16286 raft_consensus.cc:374] T 39ab9790d8db47e99f5ce2025ec30899 P e533016cc7bf4df28e3548b9bcf335f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:54:56.624462 16287 raft_consensus.cc:729] T 39ab9790d8db47e99f5ce2025ec30899 P 3e02e47de5f2497593ca8ebb2fabaec5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e02e47de5f2497593ca8ebb2fabaec5, State: Initialized, Role: FOLLOWER
I20260929 10:54:56.624475 16286 raft_consensus.cc:729] T 39ab9790d8db47e99f5ce2025ec30899 P e533016cc7bf4df28e3548b9bcf335f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e533016cc7bf4df28e3548b9bcf335f7, State: Initialized, Role: FOLLOWER
I20260929 10:54:56.624588 16287 consensus_queue.cc:260] T 39ab9790d8db47e99f5ce2025ec30899 P 3e02e47de5f2497593ca8ebb2fabaec5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f9d84ce2604b2481a2622b74b69749" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34055 } } peers { permanent_uuid: "3e02e47de5f2497593ca8ebb2fabaec5" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 40819 } } peers { permanent_uuid: "e533016cc7bf4df28e3548b9bcf335f7" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 43863 } }
I20260929 10:54:56.624595 16286 consensus_queue.cc:260] T 39ab9790d8db47e99f5ce2025ec30899 P e533016cc7bf4df28e3548b9bcf335f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f9d84ce2604b2481a2622b74b69749" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34055 } } peers { permanent_uuid: "3e02e47de5f2497593ca8ebb2fabaec5" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 40819 } } peers { permanent_uuid: "e533016cc7bf4df28e3548b9bcf335f7" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 43863 } }
I20260929 10:54:56.624854 16287 ts_tablet_manager.cc:1428] T 39ab9790d8db47e99f5ce2025ec30899 P 3e02e47de5f2497593ca8ebb2fabaec5: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260929 10:54:56.624888 16286 ts_tablet_manager.cc:1428] T 39ab9790d8db47e99f5ce2025ec30899 P e533016cc7bf4df28e3548b9bcf335f7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 10:54:56.624960 16272 heartbeater.cc:503] Master 127.10.238.254:46359 was elected leader, sending a full tablet report...
I20260929 10:54:56.624985 16141 heartbeater.cc:503] Master 127.10.238.254:46359 was elected leader, sending a full tablet report...
I20260929 10:54:56.625319 16288 raft_consensus.cc:348] T 39ab9790d8db47e99f5ce2025ec30899 P 60f9d84ce2604b2481a2622b74b69749 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f9d84ce2604b2481a2622b74b69749" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34055 } } peers { permanent_uuid: "3e02e47de5f2497593ca8ebb2fabaec5" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 40819 } } peers { permanent_uuid: "e533016cc7bf4df28e3548b9bcf335f7" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 43863 } }
I20260929 10:54:56.625473 16288 raft_consensus.cc:374] T 39ab9790d8db47e99f5ce2025ec30899 P 60f9d84ce2604b2481a2622b74b69749 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:54:56.625504 16288 raft_consensus.cc:729] T 39ab9790d8db47e99f5ce2025ec30899 P 60f9d84ce2604b2481a2622b74b69749 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60f9d84ce2604b2481a2622b74b69749, State: Initialized, Role: FOLLOWER
I20260929 10:54:56.625624 16288 consensus_queue.cc:260] T 39ab9790d8db47e99f5ce2025ec30899 P 60f9d84ce2604b2481a2622b74b69749 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f9d84ce2604b2481a2622b74b69749" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34055 } } peers { permanent_uuid: "3e02e47de5f2497593ca8ebb2fabaec5" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 40819 } } peers { permanent_uuid: "e533016cc7bf4df28e3548b9bcf335f7" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 43863 } }
I20260929 10:54:56.625875 16288 ts_tablet_manager.cc:1428] T 39ab9790d8db47e99f5ce2025ec30899 P 60f9d84ce2604b2481a2622b74b69749: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 10:54:56.625921 16010 heartbeater.cc:503] Master 127.10.238.254:46359 was elected leader, sending a full tablet report...
I20260929 10:54:56.669929 16292 raft_consensus.cc:482] T 39ab9790d8db47e99f5ce2025ec30899 P 3e02e47de5f2497593ca8ebb2fabaec5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:54:56.670051 16292 raft_consensus.cc:504] T 39ab9790d8db47e99f5ce2025ec30899 P 3e02e47de5f2497593ca8ebb2fabaec5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f9d84ce2604b2481a2622b74b69749" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34055 } } peers { permanent_uuid: "3e02e47de5f2497593ca8ebb2fabaec5" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 40819 } } peers { permanent_uuid: "e533016cc7bf4df28e3548b9bcf335f7" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 43863 } }
I20260929 10:54:56.670486 16292 leader_election.cc:288] T 39ab9790d8db47e99f5ce2025ec30899 P 3e02e47de5f2497593ca8ebb2fabaec5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60f9d84ce2604b2481a2622b74b69749 (127.10.238.193:34055), e533016cc7bf4df28e3548b9bcf335f7 (127.10.238.194:43863)
I20260929 10:54:56.671929 16283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:56.670772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:54437 (local address 127.10.238.194:43863)
0929 10:54:56.670834 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:56.670836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:56.670845 (+ 9us) server_negotiation.cc:407] Connection header received
0929 10:54:56.670901 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:56.670902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:56.670909 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:56.670992 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:56.671658 (+ 666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:56.671659 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:56.671660 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:54:56.671664 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:56.671678 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:54:56.671683 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:56.671779 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:56.671835 (+ 56us) server_negotiation.cc:299] Negotiation successful
0929 10:54:56.671850 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 10:54:56.671937 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:56.670695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.194:43863 (local address 127.10.238.195:54437)
0929 10:54:56.670777 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:56.670791 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:56.670835 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:56.671008 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:56.671009 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:56.671012 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:54:56.671524 (+ 512us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:54:56.671524 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:54:56.671550 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:54:56.671565 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:54:56.671778 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:54:56.671780 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:54:56.671781 (+ 1us) client_negotiation.cc:769] Sending connection context
0929 10:54:56.671835 (+ 54us) client_negotiation.cc:240] Negotiation successful
0929 10:54:56.671851 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 10:54:56.671962 16285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:56.670772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:60039 (local address 127.10.238.193:34055)
0929 10:54:56.670820 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:56.670822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:56.670992 (+ 170us) server_negotiation.cc:407] Connection header received
0929 10:54:56.671124 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:56.671125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:56.671130 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:56.671186 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:56.671701 (+ 515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:56.671702 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:56.671702 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:54:56.671709 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:56.671728 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:54:56.671733 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:56.671821 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:56.671870 (+ 49us) server_negotiation.cc:299] Negotiation successful
0929 10:54:56.671885 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 10:54:56.672025 16295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:56.670695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:34055 (local address 127.10.238.195:60039)
0929 10:54:56.670933 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:56.670943 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:56.671013 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:56.671215 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:56.671215 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:56.671217 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:54:56.671621 (+ 404us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:54:56.671621 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:54:56.671627 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:54:56.671637 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:54:56.671821 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:54:56.671821 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:54:56.671822 (+ 1us) client_negotiation.cc:769] Sending connection context
0929 10:54:56.671873 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 10:54:56.671885 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":42,"thread_start_us":128,"threads_started":1}
I20260929 10:54:56.672371 15964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39ab9790d8db47e99f5ce2025ec30899" candidate_uuid: "3e02e47de5f2497593ca8ebb2fabaec5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60f9d84ce2604b2481a2622b74b69749" is_pre_election: true
I20260929 10:54:56.672472 16095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39ab9790d8db47e99f5ce2025ec30899" candidate_uuid: "3e02e47de5f2497593ca8ebb2fabaec5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e533016cc7bf4df28e3548b9bcf335f7" is_pre_election: true
I20260929 10:54:56.672542 15964 raft_consensus.cc:2454] T 39ab9790d8db47e99f5ce2025ec30899 P 60f9d84ce2604b2481a2622b74b69749 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e02e47de5f2497593ca8ebb2fabaec5 in term 0.
I20260929 10:54:56.672638 16095 raft_consensus.cc:2454] T 39ab9790d8db47e99f5ce2025ec30899 P e533016cc7bf4df28e3548b9bcf335f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e02e47de5f2497593ca8ebb2fabaec5 in term 0.
I20260929 10:54:56.672768 16159 leader_election.cc:302] T 39ab9790d8db47e99f5ce2025ec30899 P 3e02e47de5f2497593ca8ebb2fabaec5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e02e47de5f2497593ca8ebb2fabaec5, 60f9d84ce2604b2481a2622b74b69749; no voters:
I20260929 10:54:56.672959 16292 raft_consensus.cc:2781] T 39ab9790d8db47e99f5ce2025ec30899 P 3e02e47de5f2497593ca8ebb2fabaec5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:54:56.673023 16292 raft_consensus.cc:482] T 39ab9790d8db47e99f5ce2025ec30899 P 3e02e47de5f2497593ca8ebb2fabaec5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:54:56.673040 16292 raft_consensus.cc:3037] T 39ab9790d8db47e99f5ce2025ec30899 P 3e02e47de5f2497593ca8ebb2fabaec5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:54:56.673856 16292 raft_consensus.cc:504] T 39ab9790d8db47e99f5ce2025ec30899 P 3e02e47de5f2497593ca8ebb2fabaec5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f9d84ce2604b2481a2622b74b69749" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34055 } } peers { permanent_uuid: "3e02e47de5f2497593ca8ebb2fabaec5" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 40819 } } peers { permanent_uuid: "e533016cc7bf4df28e3548b9bcf335f7" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 43863 } }
I20260929 10:54:56.674000 16292 leader_election.cc:288] T 39ab9790d8db47e99f5ce2025ec30899 P 3e02e47de5f2497593ca8ebb2fabaec5 [CANDIDATE]: Term 1 election: Requested vote from peers 60f9d84ce2604b2481a2622b74b69749 (127.10.238.193:34055), e533016cc7bf4df28e3548b9bcf335f7 (127.10.238.194:43863)
I20260929 10:54:56.674180 16095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39ab9790d8db47e99f5ce2025ec30899" candidate_uuid: "3e02e47de5f2497593ca8ebb2fabaec5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e533016cc7bf4df28e3548b9bcf335f7"
I20260929 10:54:56.674249 16095 raft_consensus.cc:3037] T 39ab9790d8db47e99f5ce2025ec30899 P e533016cc7bf4df28e3548b9bcf335f7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:54:56.675127 16095 raft_consensus.cc:2454] T 39ab9790d8db47e99f5ce2025ec30899 P e533016cc7bf4df28e3548b9bcf335f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e02e47de5f2497593ca8ebb2fabaec5 in term 1.
I20260929 10:54:56.675267 15964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39ab9790d8db47e99f5ce2025ec30899" candidate_uuid: "3e02e47de5f2497593ca8ebb2fabaec5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60f9d84ce2604b2481a2622b74b69749"
I20260929 10:54:56.675323 16160 leader_election.cc:302] T 39ab9790d8db47e99f5ce2025ec30899 P 3e02e47de5f2497593ca8ebb2fabaec5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e02e47de5f2497593ca8ebb2fabaec5, e533016cc7bf4df28e3548b9bcf335f7; no voters:
I20260929 10:54:56.675345 15964 raft_consensus.cc:3037] T 39ab9790d8db47e99f5ce2025ec30899 P 60f9d84ce2604b2481a2622b74b69749 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:54:56.675457 16292 raft_consensus.cc:2781] T 39ab9790d8db47e99f5ce2025ec30899 P 3e02e47de5f2497593ca8ebb2fabaec5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:54:56.675644 16292 raft_consensus.cc:686] T 39ab9790d8db47e99f5ce2025ec30899 P 3e02e47de5f2497593ca8ebb2fabaec5 [term 1 LEADER]: Becoming Leader. State: Replica: 3e02e47de5f2497593ca8ebb2fabaec5, State: Running, Role: LEADER
I20260929 10:54:56.675767 16292 consensus_queue.cc:237] T 39ab9790d8db47e99f5ce2025ec30899 P 3e02e47de5f2497593ca8ebb2fabaec5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f9d84ce2604b2481a2622b74b69749" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34055 } } peers { permanent_uuid: "3e02e47de5f2497593ca8ebb2fabaec5" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 40819 } } peers { permanent_uuid: "e533016cc7bf4df28e3548b9bcf335f7" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 43863 } }
I20260929 10:54:56.676273 15964 raft_consensus.cc:2454] T 39ab9790d8db47e99f5ce2025ec30899 P 60f9d84ce2604b2481a2622b74b69749 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e02e47de5f2497593ca8ebb2fabaec5 in term 1.
I20260929 10:54:56.676829 15822 catalog_manager.cc:5495] T 39ab9790d8db47e99f5ce2025ec30899 P 3e02e47de5f2497593ca8ebb2fabaec5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e02e47de5f2497593ca8ebb2fabaec5 (127.10.238.195). New cstate: current_term: 1 leader_uuid: "3e02e47de5f2497593ca8ebb2fabaec5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f9d84ce2604b2481a2622b74b69749" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e02e47de5f2497593ca8ebb2fabaec5" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 40819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e533016cc7bf4df28e3548b9bcf335f7" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 43863 } health_report { overall_health: UNKNOWN } } }
I20260929 10:54:56.689025 16284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:56.688024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39256 (local address 127.10.238.195:40819)
0929 10:54:56.688091 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:56.688093 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:56.688102 (+ 9us) server_negotiation.cc:407] Connection header received
0929 10:54:56.688149 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:56.688149 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:56.688162 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:56.688226 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:56.688759 (+ 533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:56.688760 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:56.688760 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:54:56.688765 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:56.688787 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:54:56.688792 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:56.688860 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:56.688931 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 10:54:56.688947 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 10:54:56.690822 15964 raft_consensus.cc:1261] T 39ab9790d8db47e99f5ce2025ec30899 P 60f9d84ce2604b2481a2622b74b69749 [term 1 FOLLOWER]: Refusing update from remote peer 3e02e47de5f2497593ca8ebb2fabaec5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:54:56.690824 16095 raft_consensus.cc:1261] T 39ab9790d8db47e99f5ce2025ec30899 P e533016cc7bf4df28e3548b9bcf335f7 [term 1 FOLLOWER]: Refusing update from remote peer 3e02e47de5f2497593ca8ebb2fabaec5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:54:56.691176 16292 consensus_queue.cc:1035] T 39ab9790d8db47e99f5ce2025ec30899 P 3e02e47de5f2497593ca8ebb2fabaec5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e533016cc7bf4df28e3548b9bcf335f7" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:54:56.691306 16296 consensus_queue.cc:1035] T 39ab9790d8db47e99f5ce2025ec30899 P 3e02e47de5f2497593ca8ebb2fabaec5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60f9d84ce2604b2481a2622b74b69749" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:54:56.694280 16300 mvcc.cc:205] Tried to move back new op lower bound from 7334622399242825728 to 7334622399184699392. Current Snapshot: MvccSnapshot[applied={T|T < 7334622399242825728}]
I20260929 10:54:56.694598 16299 mvcc.cc:205] Tried to move back new op lower bound from 7334622399242825728 to 7334622399184699392. Current Snapshot: MvccSnapshot[applied={T|T < 7334622399242825728}]
I20260929 10:54:56.695545 16301 mvcc.cc:205] Tried to move back new op lower bound from 7334622399242825728 to 7334622399184699392. Current Snapshot: MvccSnapshot[applied={T|T < 7334622399242825728 or (T in {7334622399242825728})}]
I20260929 10:54:56.696619 15822 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:54:56.696705 15822 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:54:56.697474 15822 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8e9d9278896248e790f55bd1ec081f46: Sending DeleteTablet for 3 replicas of tablet 39ab9790d8db47e99f5ce2025ec30899
I20260929 10:54:56.697785 16206 tablet_service.cc:1514] Processing DeleteTablet for tablet 39ab9790d8db47e99f5ce2025ec30899 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:54:56 UTC) from {username='slave'} at 127.0.0.1:39248
I20260929 10:54:56.697808 16075 tablet_service.cc:1514] Processing DeleteTablet for tablet 39ab9790d8db47e99f5ce2025ec30899 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:54:56 UTC) from {username='slave'} at 127.0.0.1:46746
I20260929 10:54:56.698010 16308 tablet_replica.cc:330] T 39ab9790d8db47e99f5ce2025ec30899 P 3e02e47de5f2497593ca8ebb2fabaec5: stopping tablet replica
I20260929 10:54:56.698115 16308 raft_consensus.cc:2229] T 39ab9790d8db47e99f5ce2025ec30899 P 3e02e47de5f2497593ca8ebb2fabaec5 [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:54:56.698244 16308 raft_consensus.cc:2258] T 39ab9790d8db47e99f5ce2025ec30899 P 3e02e47de5f2497593ca8ebb2fabaec5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:54:56.698668 16308 ts_tablet_manager.cc:1905] T 39ab9790d8db47e99f5ce2025ec30899 P 3e02e47de5f2497593ca8ebb2fabaec5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:54:56.699159 15944 tablet_service.cc:1514] Processing DeleteTablet for tablet 39ab9790d8db47e99f5ce2025ec30899 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:54:56 UTC) from {username='slave'} at 127.0.0.1:40184
I20260929 10:54:56.699335 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 15882
I20260929 10:54:56.701100 16308 ts_tablet_manager.cc:1918] T 39ab9790d8db47e99f5ce2025ec30899 P 3e02e47de5f2497593ca8ebb2fabaec5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:54:56.701228 16308 log.cc:1196] T 39ab9790d8db47e99f5ce2025ec30899 P 3e02e47de5f2497593ca8ebb2fabaec5: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790679274510869-11195-0/minicluster-data/ts-2/wal/wals/39ab9790d8db47e99f5ce2025ec30899
I20260929 10:54:56.701577 16308 ts_tablet_manager.cc:1939] T 39ab9790d8db47e99f5ce2025ec30899 P 3e02e47de5f2497593ca8ebb2fabaec5: Deleting consensus metadata
I20260929 10:54:56.702237 16309 tablet_replica.cc:330] T 39ab9790d8db47e99f5ce2025ec30899 P e533016cc7bf4df28e3548b9bcf335f7: stopping tablet replica
I20260929 10:54:56.702379 16309 raft_consensus.cc:2229] T 39ab9790d8db47e99f5ce2025ec30899 P e533016cc7bf4df28e3548b9bcf335f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:54:56.702484 16309 raft_consensus.cc:2258] T 39ab9790d8db47e99f5ce2025ec30899 P e533016cc7bf4df28e3548b9bcf335f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:54:56.702656 15809 catalog_manager.cc:4841] TS 3e02e47de5f2497593ca8ebb2fabaec5 (127.10.238.195:40819): tablet 39ab9790d8db47e99f5ce2025ec30899 (table test-table [id=a08253cbcfa74c6ca6261b5848fcd7e6]) successfully deleted
I20260929 10:54:56.705646 16309 ts_tablet_manager.cc:1905] T 39ab9790d8db47e99f5ce2025ec30899 P e533016cc7bf4df28e3548b9bcf335f7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 10:54:56.706791 15806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.238.193:34055 (error 108)
I20260929 10:54:56.707039 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 16013
I20260929 10:54:56.707265 16280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:56.707136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:34055 (local address 127.0.0.1:40196)
0929 10:54:56.707201 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:56.707235 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.193:34055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 10:54:56.707438 15806 catalog_manager.cc:4584] TS 60f9d84ce2604b2481a2622b74b69749 (127.10.238.193:34055): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 39ab9790d8db47e99f5ce2025ec30899: Network error: Client connection negotiation failed: client connection to 127.10.238.193:34055: connect: Connection refused (error 111)
I20260929 10:54:56.712522 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 16144
I20260929 10:54:56.712551 16280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:56.712430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.194:43863 (local address 127.0.0.1:46748)
0929 10:54:56.712487 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:56.712518 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.194:43863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 10:54:56.716717 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 15791
2026-09-29T10:54:56Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (648 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 10:54:56.736961 11195 mini_oidc.cc:152] Starting JWKS server
I20260929 10:54:56.738129 11195 webserver.cc:460] Webserver started at https://127.0.0.1:32823/ using document root <none> and password file <none>
I20260929 10:54:56.738242 11195 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 10:54:56.738420 11195 webserver.cc:460] Webserver started at http://127.0.0.1:40401/ using document root <none> and password file <none>
2026-09-29T10:54:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:54:56Z Disabled control of system clock
I20260929 10:54:56.746158 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:38559
--webserver_interface=127.10.238.254
--webserver_port=0
--builtin_ntp_servers=127.10.238.212:36991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:38559
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32823/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790679274510869-11195-0/testchainca.cert with env {}
W20260929 10:54:56.831301 16317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:54:56.831494 16317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:54:56.831517 16317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:54:56.832679 16317 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:54:56.832724 16317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:54:56.832738 16317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:54:56.832774 16317 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:54:56.832810 16317 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 10:54:56.832826 16317 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 10:54:56.832849 16317 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:32823/jwks.json
I20260929 10:54:56.834414 16317 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:36991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790679274510869-11195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:38559
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:38559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32823/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.238.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790679274510869-11195-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:54:56.834661 16317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:54:56.834868 16317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:54:56.839895 16325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:56.839902 16317 server_base.cc:1034] running on GCE node
W20260929 10:54:56.839884 16322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:56.839884 16323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:56.840329 16317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:54:56.840621 16317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:54:56.841782 16317 hybrid_clock.cc:648] HybridClock initialized: now 1790679296841757 us; error 36 us; skew 500 ppm
I20260929 10:54:56.842939 16317 webserver.cc:460] Webserver started at http://127.10.238.254:36133/ using document root <none> and password file <none>
I20260929 10:54:56.843169 16317 fs_manager.cc:359] Metadata directory not provided
I20260929 10:54:56.843237 16317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:54:56.843400 16317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:54:56.844692 16317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790679274510869-11195-0/minicluster-data/master-0/data/instance:
uuid: "38b21a094dee478f9ea4c724151f224e"
format_stamp: "Formatted at 2026-09-29 10:54:56 on dist-test-slave-8nrt"
server_key: "ed3d2a66a3e2c63d1c850ebc5eac90e7"
server_key_iv: "682c4d4bf00ed67d9bc7035029408282"
server_key_version: "encryptionkey@0"
I20260929 10:54:56.845091 16317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790679274510869-11195-0/minicluster-data/master-0/wal/instance:
uuid: "38b21a094dee478f9ea4c724151f224e"
format_stamp: "Formatted at 2026-09-29 10:54:56 on dist-test-slave-8nrt"
server_key: "ed3d2a66a3e2c63d1c850ebc5eac90e7"
server_key_iv: "682c4d4bf00ed67d9bc7035029408282"
server_key_version: "encryptionkey@0"
I20260929 10:54:56.846401 16317 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 10:54:56.847136 16331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:56.847343 16317 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 10:54:56.847440 16317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "38b21a094dee478f9ea4c724151f224e"
format_stamp: "Formatted at 2026-09-29 10:54:56 on dist-test-slave-8nrt"
server_key: "ed3d2a66a3e2c63d1c850ebc5eac90e7"
server_key_iv: "682c4d4bf00ed67d9bc7035029408282"
server_key_version: "encryptionkey@0"
I20260929 10:54:56.847527 16317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:54:56.863462 16317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:54:56.863787 16317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:54:56.863934 16317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:54:56.867952 16317 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.254:38559
I20260929 10:54:56.867987 16383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.254:38559 every 8 connection(s)
I20260929 10:54:56.868309 16317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:54:56.869191 16384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:54:56.870268 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 16317
I20260929 10:54:56.870379 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790679274510869-11195-0/minicluster-data/master-0/wal/instance
I20260929 10:54:56.870563 11195 external_mini_cluster.cc:1428] Setting key c717004c89c8ec1736af24967486bacd
I20260929 10:54:56.871927 16384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38b21a094dee478f9ea4c724151f224e: Bootstrap starting.
I20260929 10:54:56.872653 16384 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38b21a094dee478f9ea4c724151f224e: Neither blocks nor log segments found. Creating new log.
I20260929 10:54:56.872960 16384 log.cc:824] T 00000000000000000000000000000000 P 38b21a094dee478f9ea4c724151f224e: Log is configured to *not* fsync() on all Append() calls
I20260929 10:54:56.873692 16384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38b21a094dee478f9ea4c724151f224e: No bootstrap required, opened a new log
I20260929 10:54:56.876034 16384 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38b21a094dee478f9ea4c724151f224e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b21a094dee478f9ea4c724151f224e" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 38559 } }
I20260929 10:54:56.876205 16384 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38b21a094dee478f9ea4c724151f224e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:54:56.876242 16384 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38b21a094dee478f9ea4c724151f224e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38b21a094dee478f9ea4c724151f224e, State: Initialized, Role: FOLLOWER
I20260929 10:54:56.876410 16384 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38b21a094dee478f9ea4c724151f224e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b21a094dee478f9ea4c724151f224e" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 38559 } }
I20260929 10:54:56.876495 16384 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38b21a094dee478f9ea4c724151f224e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:54:56.876528 16384 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38b21a094dee478f9ea4c724151f224e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:54:56.876565 16387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:56.871052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37712 (local address 127.10.238.254:38559)
0929 10:54:56.871262 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:56.871271 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:56.871304 (+ 33us) server_negotiation.cc:407] Connection header received
0929 10:54:56.871826 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:56.871834 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:56.871909 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:56.872057 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:56.872236 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:56.873591 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:56.874400 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:56.875034 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:56.875152 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:56.875850 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:56.875854 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:56.875854 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:54:56.876055 (+ 201us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:56.876097 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:54:56.876111 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:56.876183 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:56.876277 (+ 94us) server_negotiation.cc:299] Negotiation successful
0929 10:54:56.876365 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":76,"threads_started":1}
I20260929 10:54:56.876582 16384 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38b21a094dee478f9ea4c724151f224e [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:54:56.877225 16384 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38b21a094dee478f9ea4c724151f224e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b21a094dee478f9ea4c724151f224e" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 38559 } }
I20260929 10:54:56.877352 16384 leader_election.cc:302] T 00000000000000000000000000000000 P 38b21a094dee478f9ea4c724151f224e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38b21a094dee478f9ea4c724151f224e; no voters:
I20260929 10:54:56.877514 16384 leader_election.cc:288] T 00000000000000000000000000000000 P 38b21a094dee478f9ea4c724151f224e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:54:56.877601 11195 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:54:56.877606 16389 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38b21a094dee478f9ea4c724151f224e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:54:56.877765 16384 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38b21a094dee478f9ea4c724151f224e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:54:56.877817 16389 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38b21a094dee478f9ea4c724151f224e [term 1 LEADER]: Becoming Leader. State: Replica: 38b21a094dee478f9ea4c724151f224e, State: Running, Role: LEADER
I20260929 10:54:56.877904 16389 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38b21a094dee478f9ea4c724151f224e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b21a094dee478f9ea4c724151f224e" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 38559 } }
I20260929 10:54:56.878199 16390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38b21a094dee478f9ea4c724151f224e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38b21a094dee478f9ea4c724151f224e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b21a094dee478f9ea4c724151f224e" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 38559 } } }
I20260929 10:54:56.878304 16390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38b21a094dee478f9ea4c724151f224e [sys.catalog]: This master's current role is: LEADER
I20260929 10:54:56.878201 16391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38b21a094dee478f9ea4c724151f224e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38b21a094dee478f9ea4c724151f224e. Latest consensus state: current_term: 1 leader_uuid: "38b21a094dee478f9ea4c724151f224e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b21a094dee478f9ea4c724151f224e" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 38559 } } }
I20260929 10:54:56.878528 16391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38b21a094dee478f9ea4c724151f224e [sys.catalog]: This master's current role is: LEADER
I20260929 10:54:56.878813 16396 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:54:56.879367 16396 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:54:56.881047 16396 catalog_manager.cc:1347] Generated new cluster ID: 77ebe8a0a53040b2b1462050bc71bc07
I20260929 10:54:56.881099 16396 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:54:56.888913 16396 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:54:56.889622 16396 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:54:56.897377 16396 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38b21a094dee478f9ea4c724151f224e: Generated new TSK 0
I20260929 10:54:56.897541 16396 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 10:54:56.920207 16414 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 10:54:56.920365 16414 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260929 10:54:56.920795 16413 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:54:56.904135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:38559 (local address 127.0.0.1:37726)
0929 10:54:56.904320 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:56.904330 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:56.904411 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:56.904617 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:56.904618 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:56.904620 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0929 10:54:56.904698 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:56.904700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:56.905415 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:56.905416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:56.906005 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:56.906007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:56.906386 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:56.906387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:56.906461 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:56.906478 (+ 17us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 10:54:56.920477 (+ 13999us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32823/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 10:54:56.920564 (+ 87us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.238.254:38559: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32823/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":100,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.10.238.254:38559: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32823/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 10:54:56.921334 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 16317
I20260929 10:54:56.921375 16387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:56.904220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37726 (local address 127.10.238.254:38559)
0929 10:54:56.904266 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:56.904268 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:56.904391 (+ 123us) server_negotiation.cc:407] Connection header received
0929 10:54:56.904511 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:56.904512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:56.904526 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:56.904608 (+ 82us) server_negotiation.cc:226] Negotiated authn=JWT
0929 10:54:56.904776 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:56.905300 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:56.906101 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:56.906308 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:56.906384 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:56.906585 (+ 201us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 10:54:56.920338 (+ 13753us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32823/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 10:54:56.921121 (+ 783us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37726: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32823/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":25}
2026-09-29T10:54:56Z chronyd exiting
I20260929 10:54:56.956948 11195 test_util.cc:175] -----------------------------------------------
I20260929 10:54:56.957024 11195 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790679274510869-11195-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (221 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 10:54:56.957841 11195 mini_oidc.cc:152] Starting JWKS server
I20260929 10:54:56.958458 11195 webserver.cc:460] Webserver started at https://127.0.0.1:37327/ using document root <none> and password file <none>
I20260929 10:54:56.958498 11195 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 10:54:56.958598 11195 webserver.cc:460] Webserver started at http://127.0.0.1:38223/ using document root <none> and password file <none>
2026-09-29T10:54:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:54:56Z Disabled control of system clock
I20260929 10:54:56.966145 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:38137
--webserver_interface=127.10.238.254
--webserver_port=0
--builtin_ntp_servers=127.10.238.212:36251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:38137
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37327/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790679274510869-11195-0/testchainca.cert with env {}
W20260929 10:54:57.043787 16420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:54:57.044016 16420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:54:57.044054 16420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:54:57.045202 16420 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:54:57.045303 16420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:54:57.045337 16420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:54:57.045362 16420 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:54:57.045387 16420 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 10:54:57.045430 16420 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 10:54:57.045455 16420 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37327/jwks.json
I20260929 10:54:57.046895 16420 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:36251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790679274510869-11195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:38137
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:38137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37327/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.238.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790679274510869-11195-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:54:57.047183 16420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:54:57.047397 16420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:54:57.052304 16428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:57.052331 16426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:57.052492 16425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:57.052613 16420 server_base.cc:1034] running on GCE node
I20260929 10:54:57.052870 16420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:54:57.053145 16420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:54:57.054307 16420 hybrid_clock.cc:648] HybridClock initialized: now 1790679297054300 us; error 38 us; skew 500 ppm
I20260929 10:54:57.055451 16420 webserver.cc:460] Webserver started at http://127.10.238.254:44319/ using document root <none> and password file <none>
I20260929 10:54:57.055699 16420 fs_manager.cc:359] Metadata directory not provided
I20260929 10:54:57.055774 16420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:54:57.055922 16420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:54:57.056941 16420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790679274510869-11195-0/minicluster-data/master-0/data/instance:
uuid: "8aef7761fc1a476c88ff99c5ec49864d"
format_stamp: "Formatted at 2026-09-29 10:54:57 on dist-test-slave-8nrt"
server_key: "cba656049ccbca1b998c0daf838a85cb"
server_key_iv: "dfd00e76b337a10f411832ac58fd89f7"
server_key_version: "encryptionkey@0"
I20260929 10:54:57.057359 16420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790679274510869-11195-0/minicluster-data/master-0/wal/instance:
uuid: "8aef7761fc1a476c88ff99c5ec49864d"
format_stamp: "Formatted at 2026-09-29 10:54:57 on dist-test-slave-8nrt"
server_key: "cba656049ccbca1b998c0daf838a85cb"
server_key_iv: "dfd00e76b337a10f411832ac58fd89f7"
server_key_version: "encryptionkey@0"
I20260929 10:54:57.058581 16420 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 10:54:57.059304 16434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:57.059468 16420 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 10:54:57.059564 16420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "8aef7761fc1a476c88ff99c5ec49864d"
format_stamp: "Formatted at 2026-09-29 10:54:57 on dist-test-slave-8nrt"
server_key: "cba656049ccbca1b998c0daf838a85cb"
server_key_iv: "dfd00e76b337a10f411832ac58fd89f7"
server_key_version: "encryptionkey@0"
I20260929 10:54:57.059646 16420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:54:57.103099 16420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:54:57.103473 16420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:54:57.103643 16420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:54:57.107932 16420 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.254:38137
I20260929 10:54:57.107969 16486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.254:38137 every 8 connection(s)
I20260929 10:54:57.108352 16420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:54:57.109025 16487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:54:57.110601 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 16420
I20260929 10:54:57.110710 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790679274510869-11195-0/minicluster-data/master-0/wal/instance
I20260929 10:54:57.110884 11195 external_mini_cluster.cc:1428] Setting key e18c7c2eb6e1e031b3a62785a9a0afe1
I20260929 10:54:57.112408 16487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8aef7761fc1a476c88ff99c5ec49864d: Bootstrap starting.
I20260929 10:54:57.113075 16487 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8aef7761fc1a476c88ff99c5ec49864d: Neither blocks nor log segments found. Creating new log.
I20260929 10:54:57.113411 16487 log.cc:824] T 00000000000000000000000000000000 P 8aef7761fc1a476c88ff99c5ec49864d: Log is configured to *not* fsync() on all Append() calls
I20260929 10:54:57.114246 16487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8aef7761fc1a476c88ff99c5ec49864d: No bootstrap required, opened a new log
I20260929 10:54:57.116605 16487 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8aef7761fc1a476c88ff99c5ec49864d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aef7761fc1a476c88ff99c5ec49864d" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 38137 } }
I20260929 10:54:57.116760 16487 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8aef7761fc1a476c88ff99c5ec49864d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:54:57.116783 16487 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8aef7761fc1a476c88ff99c5ec49864d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8aef7761fc1a476c88ff99c5ec49864d, State: Initialized, Role: FOLLOWER
I20260929 10:54:57.116895 16490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:57.111410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52260 (local address 127.10.238.254:38137)
0929 10:54:57.111610 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:57.111617 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:57.111642 (+ 25us) server_negotiation.cc:407] Connection header received
0929 10:54:57.112229 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:57.112236 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:57.112299 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:57.112416 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:57.112657 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:57.113838 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:57.114921 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:57.115333 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:57.115433 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:57.116157 (+ 724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:57.116161 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:57.116161 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:54:57.116332 (+ 171us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:57.116377 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:54:57.116388 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:57.116435 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:57.116534 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 10:54:57.116621 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":94,"threads_started":1}
I20260929 10:54:57.116955 16487 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8aef7761fc1a476c88ff99c5ec49864d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aef7761fc1a476c88ff99c5ec49864d" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 38137 } }
I20260929 10:54:57.117048 16487 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8aef7761fc1a476c88ff99c5ec49864d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:54:57.117087 16487 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8aef7761fc1a476c88ff99c5ec49864d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:54:57.117125 16487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8aef7761fc1a476c88ff99c5ec49864d [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:54:57.117767 16487 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8aef7761fc1a476c88ff99c5ec49864d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aef7761fc1a476c88ff99c5ec49864d" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 38137 } }
I20260929 10:54:57.117874 11195 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:54:57.117890 16487 leader_election.cc:302] T 00000000000000000000000000000000 P 8aef7761fc1a476c88ff99c5ec49864d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8aef7761fc1a476c88ff99c5ec49864d; no voters:
I20260929 10:54:57.118072 16487 leader_election.cc:288] T 00000000000000000000000000000000 P 8aef7761fc1a476c88ff99c5ec49864d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:54:57.118109 16492 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8aef7761fc1a476c88ff99c5ec49864d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:54:57.118211 16492 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8aef7761fc1a476c88ff99c5ec49864d [term 1 LEADER]: Becoming Leader. State: Replica: 8aef7761fc1a476c88ff99c5ec49864d, State: Running, Role: LEADER
I20260929 10:54:57.118299 16492 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8aef7761fc1a476c88ff99c5ec49864d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aef7761fc1a476c88ff99c5ec49864d" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 38137 } }
I20260929 10:54:57.118333 16487 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8aef7761fc1a476c88ff99c5ec49864d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:54:57.118594 16493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8aef7761fc1a476c88ff99c5ec49864d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8aef7761fc1a476c88ff99c5ec49864d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aef7761fc1a476c88ff99c5ec49864d" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 38137 } } }
I20260929 10:54:57.118703 16493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8aef7761fc1a476c88ff99c5ec49864d [sys.catalog]: This master's current role is: LEADER
I20260929 10:54:57.118985 16498 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:54:57.119014 16494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8aef7761fc1a476c88ff99c5ec49864d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8aef7761fc1a476c88ff99c5ec49864d. Latest consensus state: current_term: 1 leader_uuid: "8aef7761fc1a476c88ff99c5ec49864d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aef7761fc1a476c88ff99c5ec49864d" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 38137 } } }
I20260929 10:54:57.119083 16494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8aef7761fc1a476c88ff99c5ec49864d [sys.catalog]: This master's current role is: LEADER
I20260929 10:54:57.119525 16498 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:54:57.121084 16498 catalog_manager.cc:1347] Generated new cluster ID: 3dd913a4e62d414c927dd762affafe20
I20260929 10:54:57.121152 16498 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:54:57.138550 16498 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:54:57.139103 16498 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:54:57.144052 16498 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8aef7761fc1a476c88ff99c5ec49864d: Generated new TSK 0
I20260929 10:54:57.144248 16498 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 10:54:57.170504 16517 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 10:54:57.170818 16517 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 10:54:57.171124 16490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:57.155953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52274 (local address 127.10.238.254:38137)
0929 10:54:57.156013 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:57.156015 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:57.156112 (+ 97us) server_negotiation.cc:407] Connection header received
0929 10:54:57.156167 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:57.156168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:57.156180 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:57.156249 (+ 69us) server_negotiation.cc:226] Negotiated authn=JWT
0929 10:54:57.156382 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:57.157176 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:57.158049 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:57.158363 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:57.158438 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:57.158603 (+ 165us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 10:54:57.170712 (+ 12109us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37327/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 10:54:57.171050 (+ 338us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52274: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37327/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":36}
W20260929 10:54:57.171180 16490 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52274: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37327/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260929 10:54:57.171211 16516 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:54:57.155868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:38137 (local address 127.0.0.1:52274)
0929 10:54:57.156035 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:57.156046 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:57.156105 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:57.156257 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:57.156258 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:57.156260 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0929 10:54:57.156306 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:57.156309 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:57.157306 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:57.157306 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:57.157967 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:57.157969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:57.158444 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:57.158445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:57.158484 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:57.158489 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 10:54:57.170993 (+ 12504us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37327/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 10:54:57.171094 (+ 101us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.238.254:38137: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37327/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":98,"threads_started":1}
I20260929 10:54:57.171453 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 16420
2026-09-29T10:54:57Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (251 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 10:54:57.209364 11195 mini_oidc.cc:152] Starting JWKS server
I20260929 10:54:57.210034 11195 webserver.cc:460] Webserver started at https://127.0.0.1:45193/ using document root <none> and password file <none>
I20260929 10:54:57.210073 11195 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 10:54:57.210179 11195 webserver.cc:460] Webserver started at http://127.0.0.1:35281/ using document root <none> and password file <none>
2026-09-29T10:54:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:54:57Z Disabled control of system clock
I20260929 10:54:57.217777 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:42469
--webserver_interface=127.10.238.254
--webserver_port=0
--builtin_ntp_servers=127.10.238.212:33299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:42469
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45193/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 10:54:57.297706 16523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:54:57.297897 16523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:54:57.297917 16523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:54:57.299033 16523 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:54:57.299069 16523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:54:57.299082 16523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:54:57.299095 16523 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:54:57.299108 16523 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 10:54:57.299119 16523 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 10:54:57.299139 16523 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45193/jwks.json
I20260929 10:54:57.300405 16523 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:33299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790679274510869-11195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:42469
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:42469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45193/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:54:57.300580 16523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:54:57.300814 16523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:54:57.305634 16528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:57.305603 16531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:57.305609 16529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:57.305879 16523 server_base.cc:1034] running on GCE node
I20260929 10:54:57.306102 16523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:54:57.306347 16523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:54:57.307497 16523 hybrid_clock.cc:648] HybridClock initialized: now 1790679297307479 us; error 29 us; skew 500 ppm
I20260929 10:54:57.309113 16523 webserver.cc:460] Webserver started at http://127.10.238.254:38617/ using document root <none> and password file <none>
I20260929 10:54:57.309435 16523 fs_manager.cc:359] Metadata directory not provided
I20260929 10:54:57.309513 16523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:54:57.309677 16523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:54:57.310776 16523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790679274510869-11195-0/minicluster-data/master-0/data/instance:
uuid: "fa3ee4aaa30042e1b3e26f4bf0511a22"
format_stamp: "Formatted at 2026-09-29 10:54:57 on dist-test-slave-8nrt"
server_key: "c4ce0bfb904888d070b355e15af6492d"
server_key_iv: "d6c49ae8e45b59379c4e0ef886cbc02d"
server_key_version: "encryptionkey@0"
I20260929 10:54:57.311151 16523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790679274510869-11195-0/minicluster-data/master-0/wal/instance:
uuid: "fa3ee4aaa30042e1b3e26f4bf0511a22"
format_stamp: "Formatted at 2026-09-29 10:54:57 on dist-test-slave-8nrt"
server_key: "c4ce0bfb904888d070b355e15af6492d"
server_key_iv: "d6c49ae8e45b59379c4e0ef886cbc02d"
server_key_version: "encryptionkey@0"
I20260929 10:54:57.312453 16523 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 10:54:57.313225 16537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:57.313450 16523 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:54:57.313553 16523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "fa3ee4aaa30042e1b3e26f4bf0511a22"
format_stamp: "Formatted at 2026-09-29 10:54:57 on dist-test-slave-8nrt"
server_key: "c4ce0bfb904888d070b355e15af6492d"
server_key_iv: "d6c49ae8e45b59379c4e0ef886cbc02d"
server_key_version: "encryptionkey@0"
I20260929 10:54:57.313645 16523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:54:57.328895 16523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:54:57.329237 16523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:54:57.329460 16523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:54:57.333616 16523 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.254:42469
I20260929 10:54:57.333659 16589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.254:42469 every 8 connection(s)
I20260929 10:54:57.334012 16523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:54:57.334673 16590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:54:57.337456 16590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa3ee4aaa30042e1b3e26f4bf0511a22: Bootstrap starting.
I20260929 10:54:57.338032 16590 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa3ee4aaa30042e1b3e26f4bf0511a22: Neither blocks nor log segments found. Creating new log.
I20260929 10:54:57.338325 16590 log.cc:824] T 00000000000000000000000000000000 P fa3ee4aaa30042e1b3e26f4bf0511a22: Log is configured to *not* fsync() on all Append() calls
I20260929 10:54:57.338980 16590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa3ee4aaa30042e1b3e26f4bf0511a22: No bootstrap required, opened a new log
I20260929 10:54:57.340580 16590 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa3ee4aaa30042e1b3e26f4bf0511a22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa3ee4aaa30042e1b3e26f4bf0511a22" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 42469 } }
I20260929 10:54:57.340714 16590 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa3ee4aaa30042e1b3e26f4bf0511a22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:54:57.340736 16590 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa3ee4aaa30042e1b3e26f4bf0511a22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa3ee4aaa30042e1b3e26f4bf0511a22, State: Initialized, Role: FOLLOWER
I20260929 10:54:57.340925 16590 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa3ee4aaa30042e1b3e26f4bf0511a22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa3ee4aaa30042e1b3e26f4bf0511a22" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 42469 } }
I20260929 10:54:57.340989 16590 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa3ee4aaa30042e1b3e26f4bf0511a22 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:54:57.341010 16590 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa3ee4aaa30042e1b3e26f4bf0511a22 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:54:57.341080 16590 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa3ee4aaa30042e1b3e26f4bf0511a22 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:54:57.341676 16590 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa3ee4aaa30042e1b3e26f4bf0511a22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa3ee4aaa30042e1b3e26f4bf0511a22" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 42469 } }
I20260929 10:54:57.341789 16590 leader_election.cc:302] T 00000000000000000000000000000000 P fa3ee4aaa30042e1b3e26f4bf0511a22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa3ee4aaa30042e1b3e26f4bf0511a22; no voters:
I20260929 10:54:57.341958 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 16523
I20260929 10:54:57.341980 16590 leader_election.cc:288] T 00000000000000000000000000000000 P fa3ee4aaa30042e1b3e26f4bf0511a22 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:54:57.342011 16593 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa3ee4aaa30042e1b3e26f4bf0511a22 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:54:57.342080 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790679274510869-11195-0/minicluster-data/master-0/wal/instance
I20260929 10:54:57.342191 16593 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa3ee4aaa30042e1b3e26f4bf0511a22 [term 1 LEADER]: Becoming Leader. State: Replica: fa3ee4aaa30042e1b3e26f4bf0511a22, State: Running, Role: LEADER
I20260929 10:54:57.342245 16590 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa3ee4aaa30042e1b3e26f4bf0511a22 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:54:57.342269 11195 external_mini_cluster.cc:1428] Setting key eee421d1ba62a2fa5a997fcb70dc6307
I20260929 10:54:57.342296 16593 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa3ee4aaa30042e1b3e26f4bf0511a22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa3ee4aaa30042e1b3e26f4bf0511a22" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 42469 } }
I20260929 10:54:57.342603 16593 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa3ee4aaa30042e1b3e26f4bf0511a22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa3ee4aaa30042e1b3e26f4bf0511a22. Latest consensus state: current_term: 1 leader_uuid: "fa3ee4aaa30042e1b3e26f4bf0511a22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa3ee4aaa30042e1b3e26f4bf0511a22" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 42469 } } }
I20260929 10:54:57.342696 16593 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa3ee4aaa30042e1b3e26f4bf0511a22 [sys.catalog]: This master's current role is: LEADER
I20260929 10:54:57.342610 16595 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa3ee4aaa30042e1b3e26f4bf0511a22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa3ee4aaa30042e1b3e26f4bf0511a22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa3ee4aaa30042e1b3e26f4bf0511a22" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 42469 } } }
I20260929 10:54:57.342773 16595 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa3ee4aaa30042e1b3e26f4bf0511a22 [sys.catalog]: This master's current role is: LEADER
I20260929 10:54:57.343477 16599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:54:57.343998 16599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:54:57.345611 16599 catalog_manager.cc:1347] Generated new cluster ID: 4afbb24ccfa34ac19654952db8326673
I20260929 10:54:57.345665 16599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:54:57.348645 16604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:57.342899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49950 (local address 127.10.238.254:42469)
0929 10:54:57.343114 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:57.343122 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:57.343209 (+ 87us) server_negotiation.cc:407] Connection header received
0929 10:54:57.343382 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:57.343390 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:57.343464 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:57.343880 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:57.343990 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:57.345332 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:57.346470 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:57.347046 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:57.347454 (+ 408us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:57.347886 (+ 432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:57.347889 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:57.347890 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:54:57.348088 (+ 198us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:57.348126 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:54:57.348139 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:57.348342 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:57.348372 (+ 30us) server_negotiation.cc:299] Negotiation successful
0929 10:54:57.348444 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":76,"threads_started":1}
I20260929 10:54:57.349582 11195 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:54:57.355705 16599 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:54:57.356155 16599 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:54:57.364573 16599 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa3ee4aaa30042e1b3e26f4bf0511a22: Generated new TSK 0
I20260929 10:54:57.364739 16599 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:54:57.378978 16604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:57.375391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49954 (local address 127.10.238.254:42469)
0929 10:54:57.375449 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:57.375451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:57.375546 (+ 95us) server_negotiation.cc:407] Connection header received
0929 10:54:57.375586 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:57.375587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:57.375599 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:57.375675 (+ 76us) server_negotiation.cc:226] Negotiated authn=JWT
0929 10:54:57.375787 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:57.376583 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:57.377346 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:57.377666 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:57.377730 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:57.377883 (+ 153us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 10:54:57.378779 (+ 896us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45193/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 10:54:57.378911 (+ 132us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49954: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45193/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":37}
W20260929 10:54:57.379034 16604 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49954: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45193/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260929 10:54:57.380999 16619 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:54:57.375303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:42469 (local address 127.0.0.1:49954)
0929 10:54:57.375474 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:57.375484 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:57.375542 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:57.375678 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:57.375679 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:57.375681 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0929 10:54:57.375727 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:57.375729 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:57.376654 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:57.376654 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:57.377237 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:57.377238 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:57.377751 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:57.377751 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:57.377792 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:57.377797 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 10:54:57.378877 (+ 1080us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45193/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 10:54:57.380907 (+ 2030us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.238.254:42469: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45193/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":1838,"thread_start_us":108,"threads_started":1}
I20260929 10:54:57.381503 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 16523
E20260929 10:54:57.389904 16620 webserver.cc:533] Webserver: sslize failed:
E20260929 10:54:57.389993 16620 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-29T10:54:57Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (209 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T10:54:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:54:57Z Disabled control of system clock
I20260929 10:54:57.426069 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:37537
--webserver_interface=127.10.238.254
--webserver_port=0
--builtin_ntp_servers=127.10.238.212:37797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:37537
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790679274510869-11195-0/testchainca.cert with env {}
W20260929 10:54:57.505952 16624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:54:57.506155 16624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:54:57.506178 16624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:54:57.507354 16624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:54:57.507392 16624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:54:57.507404 16624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:54:57.507416 16624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:54:57.507428 16624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 10:54:57.507439 16624 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 10:54:57.507450 16624 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 10:54:57.508926 16624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:37797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790679274510869-11195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:37537
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:37537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.238.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790679274510869-11195-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:54:57.509184 16624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:54:57.509459 16624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:54:57.514204 16629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:57.514283 16630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:57.514412 16632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:57.514434 16624 server_base.cc:1034] running on GCE node
I20260929 10:54:57.514684 16624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:54:57.514964 16624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:54:57.516129 16624 hybrid_clock.cc:648] HybridClock initialized: now 1790679297516120 us; error 36 us; skew 500 ppm
I20260929 10:54:57.517395 16624 webserver.cc:460] Webserver started at http://127.10.238.254:38319/ using document root <none> and password file <none>
I20260929 10:54:57.517632 16624 fs_manager.cc:359] Metadata directory not provided
I20260929 10:54:57.517699 16624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:54:57.517858 16624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:54:57.518949 16624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790679274510869-11195-0/minicluster-data/master-0/data/instance:
uuid: "7dd37a919cb24aff805621f5a50b08db"
format_stamp: "Formatted at 2026-09-29 10:54:57 on dist-test-slave-8nrt"
server_key: "a5082f8c3510a298d97fbd39750c63d8"
server_key_iv: "f9da7b0180a1a1877b4ac5ecfdfc64e1"
server_key_version: "encryptionkey@0"
I20260929 10:54:57.519323 16624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790679274510869-11195-0/minicluster-data/master-0/wal/instance:
uuid: "7dd37a919cb24aff805621f5a50b08db"
format_stamp: "Formatted at 2026-09-29 10:54:57 on dist-test-slave-8nrt"
server_key: "a5082f8c3510a298d97fbd39750c63d8"
server_key_iv: "f9da7b0180a1a1877b4ac5ecfdfc64e1"
server_key_version: "encryptionkey@0"
I20260929 10:54:57.520625 16624 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 10:54:57.521356 16638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:57.521528 16624 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:54:57.521624 16624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "7dd37a919cb24aff805621f5a50b08db"
format_stamp: "Formatted at 2026-09-29 10:54:57 on dist-test-slave-8nrt"
server_key: "a5082f8c3510a298d97fbd39750c63d8"
server_key_iv: "f9da7b0180a1a1877b4ac5ecfdfc64e1"
server_key_version: "encryptionkey@0"
I20260929 10:54:57.521706 16624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:54:57.533399 16624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:54:57.533686 16624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:54:57.533846 16624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:54:57.537738 16624 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.254:37537
I20260929 10:54:57.537770 16690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.254:37537 every 8 connection(s)
I20260929 10:54:57.538093 16624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:54:57.539021 16691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:54:57.540059 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 16624
I20260929 10:54:57.540171 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790679274510869-11195-0/minicluster-data/master-0/wal/instance
I20260929 10:54:57.540373 11195 external_mini_cluster.cc:1428] Setting key 8f2205a61f3a88b2f35597135f2649f2
I20260929 10:54:57.541651 16691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dd37a919cb24aff805621f5a50b08db: Bootstrap starting.
I20260929 10:54:57.542327 16691 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7dd37a919cb24aff805621f5a50b08db: Neither blocks nor log segments found. Creating new log.
I20260929 10:54:57.542616 16691 log.cc:824] T 00000000000000000000000000000000 P 7dd37a919cb24aff805621f5a50b08db: Log is configured to *not* fsync() on all Append() calls
I20260929 10:54:57.543291 16691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dd37a919cb24aff805621f5a50b08db: No bootstrap required, opened a new log
I20260929 10:54:57.545182 16691 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7dd37a919cb24aff805621f5a50b08db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dd37a919cb24aff805621f5a50b08db" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 37537 } }
I20260929 10:54:57.545390 16691 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7dd37a919cb24aff805621f5a50b08db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:54:57.545446 16691 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7dd37a919cb24aff805621f5a50b08db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dd37a919cb24aff805621f5a50b08db, State: Initialized, Role: FOLLOWER
I20260929 10:54:57.545598 16691 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7dd37a919cb24aff805621f5a50b08db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dd37a919cb24aff805621f5a50b08db" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 37537 } }
I20260929 10:54:57.545704 16691 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7dd37a919cb24aff805621f5a50b08db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:54:57.545738 16691 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7dd37a919cb24aff805621f5a50b08db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:54:57.545795 16691 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7dd37a919cb24aff805621f5a50b08db [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:54:57.545969 16694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:57.540949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44098 (local address 127.10.238.254:37537)
0929 10:54:57.541134 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:57.541141 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:57.541167 (+ 26us) server_negotiation.cc:407] Connection header received
0929 10:54:57.541611 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:57.541617 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:57.541678 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:57.541845 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:57.542043 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:57.543002 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:57.543847 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:57.544255 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:57.544389 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:57.545132 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:57.545142 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:57.545143 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:54:57.545369 (+ 226us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:57.545401 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:54:57.545429 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:57.545649 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:57.545674 (+ 25us) server_negotiation.cc:299] Negotiation successful
0929 10:54:57.545770 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":86,"threads_started":1}
I20260929 10:54:57.546381 16691 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7dd37a919cb24aff805621f5a50b08db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dd37a919cb24aff805621f5a50b08db" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 37537 } }
I20260929 10:54:57.546506 16691 leader_election.cc:302] T 00000000000000000000000000000000 P 7dd37a919cb24aff805621f5a50b08db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7dd37a919cb24aff805621f5a50b08db; no voters:
I20260929 10:54:57.546674 16691 leader_election.cc:288] T 00000000000000000000000000000000 P 7dd37a919cb24aff805621f5a50b08db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:54:57.546763 16696 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7dd37a919cb24aff805621f5a50b08db [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:54:57.546878 16696 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7dd37a919cb24aff805621f5a50b08db [term 1 LEADER]: Becoming Leader. State: Replica: 7dd37a919cb24aff805621f5a50b08db, State: Running, Role: LEADER
I20260929 10:54:57.546933 16691 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7dd37a919cb24aff805621f5a50b08db [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:54:57.546955 11195 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:54:57.546950 16696 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7dd37a919cb24aff805621f5a50b08db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dd37a919cb24aff805621f5a50b08db" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 37537 } }
I20260929 10:54:57.547266 16698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dd37a919cb24aff805621f5a50b08db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7dd37a919cb24aff805621f5a50b08db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dd37a919cb24aff805621f5a50b08db" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 37537 } } }
I20260929 10:54:57.547362 16698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dd37a919cb24aff805621f5a50b08db [sys.catalog]: This master's current role is: LEADER
I20260929 10:54:57.547250 16696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dd37a919cb24aff805621f5a50b08db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7dd37a919cb24aff805621f5a50b08db. Latest consensus state: current_term: 1 leader_uuid: "7dd37a919cb24aff805621f5a50b08db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dd37a919cb24aff805621f5a50b08db" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 37537 } } }
I20260929 10:54:57.547542 16696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dd37a919cb24aff805621f5a50b08db [sys.catalog]: This master's current role is: LEADER
I20260929 10:54:57.547657 16702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:54:57.548313 16702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:54:57.549782 16702 catalog_manager.cc:1347] Generated new cluster ID: 63476bd8593f4738adc7e8bbbe3a1d08
I20260929 10:54:57.549834 16702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:54:57.570474 16702 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:54:57.571135 16702 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:54:57.583209 16702 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7dd37a919cb24aff805621f5a50b08db: Generated new TSK 0
I20260929 10:54:57.583413 16702 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:54:57.761760 16694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:57.588086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44110 (local address 127.10.238.254:37537)
0929 10:54:57.588141 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:57.588143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:57.588284 (+ 141us) server_negotiation.cc:407] Connection header received
0929 10:54:57.588319 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:57.588319 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:57.588331 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:57.588414 (+ 83us) server_negotiation.cc:226] Negotiated authn=JWT
0929 10:54:57.588494 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:57.589254 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:57.590058 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:57.590397 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:57.590466 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:57.590612 (+ 146us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 10:54:57.761543 (+170931us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 10:54:57.761700 (+ 157us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44110: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":33}
W20260929 10:54:57.761823 16694 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44110: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260929 10:54:57.761946 16720 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:54:57.588038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:37537 (local address 127.0.0.1:44110)
0929 10:54:57.588187 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:57.588210 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:57.588266 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:57.588410 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:57.588411 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:57.588412 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0929 10:54:57.588457 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:57.588458 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:57.589365 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:57.589366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:57.589974 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:57.589976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:57.590475 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:57.590476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:57.590515 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:57.590519 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 10:54:57.761678 (+171159us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 10:54:57.761801 (+ 123us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.238.254:37537: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":70,"threads_started":1}
I20260929 10:54:57.762269 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 16624
2026-09-29T10:54:57Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (360 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (86 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (88 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:54:57 dist-test-slave-8nrt krb5kdc[16732](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:54:57 dist-test-slave-8nrt krb5kdc[16732](info): set up 2 sockets
Sep 29 10:54:57 dist-test-slave-8nrt krb5kdc[16732](info): commencing operation
krb5kdc: starting...
W20260929 10:54:58.977505 11195 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Sep 29 10:54:59 dist-test-slave-8nrt krb5kdc[16732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:54:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:54:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
I20260929 10:54:59.095759 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:41619
--webserver_interface=127.10.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:40833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:41619
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:54:59.176273 16745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:54:59.176492 16745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:54:59.176514 16745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:54:59.177719 16745 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:54:59.177760 16745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:54:59.177774 16745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:54:59.177836 16745 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:54:59.177855 16745 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:54:59.179299 16745 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:40833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:41619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:41619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:54:59.179564 16745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:54:59.179804 16745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:54:59.184633 16750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:59.184623 16753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:59.184623 16751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:59.184933 16745 server_base.cc:1034] running on GCE node
I20260929 10:54:59.185155 16745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:54:59.185488 16745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:54:59.186651 16745 hybrid_clock.cc:648] HybridClock initialized: now 1790679299186646 us; error 42 us; skew 500 ppm
Sep 29 10:54:59 dist-test-slave-8nrt krb5kdc[16732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679299, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:54:59.189054 16745 init.cc:377] Logged in from keytab as kudu/127.10.238.254@KRBTEST.COM (short username kudu)
I20260929 10:54:59.189558 16745 webserver.cc:460] Webserver started at http://127.10.238.254:44531/ using document root <none> and password file <none>
I20260929 10:54:59.189800 16745 fs_manager.cc:359] Metadata directory not provided
I20260929 10:54:59.189846 16745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:54:59.189985 16745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:54:59.191004 16745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/master-0/data/instance:
uuid: "0b274a7dd31145a19fa00a316c845cbd"
format_stamp: "Formatted at 2026-09-29 10:54:59 on dist-test-slave-8nrt"
server_key: "febd0afadfb07785ce7a2d6b706407d7"
server_key_iv: "8030cb1067e3651b4f43916840624496"
server_key_version: "encryptionkey@0"
I20260929 10:54:59.191356 16745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/master-0/wal/instance:
uuid: "0b274a7dd31145a19fa00a316c845cbd"
format_stamp: "Formatted at 2026-09-29 10:54:59 on dist-test-slave-8nrt"
server_key: "febd0afadfb07785ce7a2d6b706407d7"
server_key_iv: "8030cb1067e3651b4f43916840624496"
server_key_version: "encryptionkey@0"
I20260929 10:54:59.192663 16745 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 10:54:59.193419 16760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:59.193596 16745 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 10:54:59.193696 16745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "0b274a7dd31145a19fa00a316c845cbd"
format_stamp: "Formatted at 2026-09-29 10:54:59 on dist-test-slave-8nrt"
server_key: "febd0afadfb07785ce7a2d6b706407d7"
server_key_iv: "8030cb1067e3651b4f43916840624496"
server_key_version: "encryptionkey@0"
I20260929 10:54:59.193778 16745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:54:59.208875 16745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:54:59.210775 16745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:54:59.210933 16745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:54:59.215027 16745 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.254:41619
I20260929 10:54:59.215059 16812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.254:41619 every 8 connection(s)
I20260929 10:54:59.215407 16745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:54:59.216250 16813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:54:59.218603 16813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b274a7dd31145a19fa00a316c845cbd: Bootstrap starting.
I20260929 10:54:59.219162 16813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b274a7dd31145a19fa00a316c845cbd: Neither blocks nor log segments found. Creating new log.
I20260929 10:54:59.219408 16813 log.cc:824] T 00000000000000000000000000000000 P 0b274a7dd31145a19fa00a316c845cbd: Log is configured to *not* fsync() on all Append() calls
I20260929 10:54:59.220019 16813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b274a7dd31145a19fa00a316c845cbd: No bootstrap required, opened a new log
I20260929 10:54:59.220274 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 16745
I20260929 10:54:59.220366 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/master-0/wal/instance
I20260929 10:54:59.220556 11195 external_mini_cluster.cc:1428] Setting key d49720d0f59a5dafe45007415a4e2dfd
I20260929 10:54:59.222193 16813 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b274a7dd31145a19fa00a316c845cbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b274a7dd31145a19fa00a316c845cbd" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 41619 } }
I20260929 10:54:59.222361 16813 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b274a7dd31145a19fa00a316c845cbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:54:59.222399 16813 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b274a7dd31145a19fa00a316c845cbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b274a7dd31145a19fa00a316c845cbd, State: Initialized, Role: FOLLOWER
I20260929 10:54:59.222584 16813 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b274a7dd31145a19fa00a316c845cbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b274a7dd31145a19fa00a316c845cbd" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 41619 } }
I20260929 10:54:59.222676 16813 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b274a7dd31145a19fa00a316c845cbd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:54:59.222725 16813 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b274a7dd31145a19fa00a316c845cbd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:54:59.222779 16813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b274a7dd31145a19fa00a316c845cbd [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:54:59.223727 16813 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b274a7dd31145a19fa00a316c845cbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b274a7dd31145a19fa00a316c845cbd" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 41619 } }
I20260929 10:54:59.223815 16813 leader_election.cc:302] T 00000000000000000000000000000000 P 0b274a7dd31145a19fa00a316c845cbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b274a7dd31145a19fa00a316c845cbd; no voters:
I20260929 10:54:59.223991 16813 leader_election.cc:288] T 00000000000000000000000000000000 P 0b274a7dd31145a19fa00a316c845cbd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:54:59.224023 16818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b274a7dd31145a19fa00a316c845cbd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:54:59.224182 16818 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b274a7dd31145a19fa00a316c845cbd [term 1 LEADER]: Becoming Leader. State: Replica: 0b274a7dd31145a19fa00a316c845cbd, State: Running, Role: LEADER
I20260929 10:54:59.224241 16813 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b274a7dd31145a19fa00a316c845cbd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:54:59.224328 16818 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b274a7dd31145a19fa00a316c845cbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b274a7dd31145a19fa00a316c845cbd" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 41619 } }
I20260929 10:54:59.224773 16819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b274a7dd31145a19fa00a316c845cbd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b274a7dd31145a19fa00a316c845cbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b274a7dd31145a19fa00a316c845cbd" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 41619 } } }
I20260929 10:54:59.224850 16819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b274a7dd31145a19fa00a316c845cbd [sys.catalog]: This master's current role is: LEADER
I20260929 10:54:59.225035 16820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b274a7dd31145a19fa00a316c845cbd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b274a7dd31145a19fa00a316c845cbd. Latest consensus state: current_term: 1 leader_uuid: "0b274a7dd31145a19fa00a316c845cbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b274a7dd31145a19fa00a316c845cbd" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 41619 } } }
I20260929 10:54:59.225126 16820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b274a7dd31145a19fa00a316c845cbd [sys.catalog]: This master's current role is: LEADER
I20260929 10:54:59.225471 16826 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:54:59.226009 16826 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 29 10:54:59 dist-test-slave-8nrt krb5kdc[16732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:54:59.229674 16826 catalog_manager.cc:1347] Generated new cluster ID: 37735bc1ca474c15b3d4867d43f664ef
I20260929 10:54:59.229729 16826 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:54:59.231969 16817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:59.221040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47154 (local address 127.10.238.254:41619)
0929 10:54:59.221226 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:59.221233 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:59.221305 (+ 72us) server_negotiation.cc:407] Connection header received
0929 10:54:59.222176 (+ 871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:59.222182 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:59.222226 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:59.222339 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:59.223160 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:59.224246 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:59.225373 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:59.226097 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:59.226205 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:59.228733 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:59.228737 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:59.228737 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:59.228749 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:59.230451 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:59.230820 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:59.230823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:59.230823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:59.230923 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:59.231130 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:59.231130 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:59.231131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:59.231371 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:59.231545 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:59.231612 (+ 67us) server_negotiation.cc:299] Negotiation successful
0929 10:54:59.231763 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":98,"threads_started":1}
I20260929 10:54:59.249765 16826 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:54:59.250252 16826 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:54:59.257607 16826 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b274a7dd31145a19fa00a316c845cbd: Generated new TSK 0
I20260929 10:54:59.257802 16826 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.238.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab.
Entry for principal kudu/127.10.238.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab.
I20260929 10:54:59.286278 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.193:0
--local_ip_for_outbound_sockets=127.10.238.193
--webserver_interface=127.10.238.193
--webserver_port=0
--tserver_master_addrs=127.10.238.254:41619
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab
--principal=kudu/127.10.238.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:40833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:54:59.366976 16839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:54:59.367195 16839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:54:59.367234 16839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:54:59.368407 16839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:54:59.368466 16839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:54:59.368511 16839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.193
I20260929 10:54:59.370023 16839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:40833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab
--principal=kudu/127.10.238.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.238.193
--webserver_port=0
--tserver_master_addrs=127.10.238.254:41619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.193
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:54:59.370287 16839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:54:59.370507 16839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:54:59.375401 16844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:59.375424 16847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:59.375586 16839 server_base.cc:1034] running on GCE node
W20260929 10:54:59.375615 16845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:59.375897 16839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:54:59.376135 16839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:54:59.377316 16839 hybrid_clock.cc:648] HybridClock initialized: now 1790679299377304 us; error 45 us; skew 500 ppm
Sep 29 10:54:59 dist-test-slave-8nrt krb5kdc[16732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679299, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:54:59.379727 16839 init.cc:377] Logged in from keytab as kudu/127.10.238.193@KRBTEST.COM (short username kudu)
I20260929 10:54:59.380159 16839 webserver.cc:460] Webserver started at http://127.10.238.193:39207/ using document root <none> and password file <none>
I20260929 10:54:59.380395 16839 fs_manager.cc:359] Metadata directory not provided
I20260929 10:54:59.380465 16839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:54:59.380620 16839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:54:59.381767 16839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/ts-0/data/instance:
uuid: "19c70c11a2e5459e8857d796f4faf503"
format_stamp: "Formatted at 2026-09-29 10:54:59 on dist-test-slave-8nrt"
server_key: "505dae63e8c06be825477d3622e1341c"
server_key_iv: "1b8d59d85d5f912890a5a9d6daa4177a"
server_key_version: "encryptionkey@0"
I20260929 10:54:59.382125 16839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/ts-0/wal/instance:
uuid: "19c70c11a2e5459e8857d796f4faf503"
format_stamp: "Formatted at 2026-09-29 10:54:59 on dist-test-slave-8nrt"
server_key: "505dae63e8c06be825477d3622e1341c"
server_key_iv: "1b8d59d85d5f912890a5a9d6daa4177a"
server_key_version: "encryptionkey@0"
I20260929 10:54:59.383432 16839 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 10:54:59.384224 16854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:59.384399 16839 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:54:59.384495 16839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/ts-0/wal
uuid: "19c70c11a2e5459e8857d796f4faf503"
format_stamp: "Formatted at 2026-09-29 10:54:59 on dist-test-slave-8nrt"
server_key: "505dae63e8c06be825477d3622e1341c"
server_key_iv: "1b8d59d85d5f912890a5a9d6daa4177a"
server_key_version: "encryptionkey@0"
I20260929 10:54:59.384582 16839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:54:59.403517 16839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:54:59.405373 16839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:54:59.405534 16839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:54:59.405771 16839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:54:59.406085 16839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:54:59.406139 16839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:59.406179 16839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:54:59.406207 16839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:59.412122 16839 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.193:40525
I20260929 10:54:59.412174 16967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.193:40525 every 8 connection(s)
I20260929 10:54:59.412474 16839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
Sep 29 10:54:59 dist-test-slave-8nrt krb5kdc[16732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679299, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.193@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:54:59.421387 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 16839
I20260929 10:54:59.421510 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/ts-0/wal/instance
I20260929 10:54:59.421550 16817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:59.413166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:55325 (local address 127.10.238.254:41619)
0929 10:54:59.413234 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:59.413236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:59.413448 (+ 212us) server_negotiation.cc:407] Connection header received
0929 10:54:59.414440 (+ 992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:59.414442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:59.414458 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:59.414511 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:59.415126 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:59.415636 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:59.416642 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:59.416865 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:59.416951 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:59.419005 (+ 2054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:59.419006 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:59.419007 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:59.419015 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:59.420201 (+ 1186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:59.420602 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:59.420603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:59.420604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:59.420662 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:59.420908 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:59.420909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:59.420909 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:59.421015 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:59.421081 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:59.421381 (+ 300us) server_negotiation.cc:299] Negotiation successful
0929 10:54:59.421464 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 10:54:59.421736 16970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:59.413148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:41619 (local address 127.10.238.193:55325)
0929 10:54:59.413321 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:59.413359 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:59.414324 (+ 965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:59.414587 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:59.414591 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:59.414840 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:54:59.415027 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:59.415031 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:59.415739 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:59.415741 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:59.416548 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:59.416550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:59.416961 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:59.416962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:59.417089 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:59.417575 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:54:59.417583 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:54:59.418897 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:54:59.420289 (+ 1392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:59.420297 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:59.420299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:59.420527 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:59.420726 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:59.420727 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:59.420728 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:59.420832 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:59.421084 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:54:59.421091 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:54:59.421225 (+ 134us) client_negotiation.cc:769] Sending connection context
0929 10:54:59.421359 (+ 134us) client_negotiation.cc:240] Negotiation successful
0929 10:54:59.421501 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":68,"threads_started":1}
I20260929 10:54:59.421751 11195 external_mini_cluster.cc:1428] Setting key 7a778449c2ea41c20f6d571c08cb1e36
I20260929 10:54:59.422308 16968 heartbeater.cc:347] Connected to a master server at 127.10.238.254:41619
I20260929 10:54:59.422410 16968 heartbeater.cc:464] Registering TS with master...
I20260929 10:54:59.422601 16968 heartbeater.cc:511] Master 127.10.238.254:41619 requested a full tablet report, sending...
I20260929 10:54:59.423182 16776 ts_manager.cc:194] Registered new tserver with Master: 19c70c11a2e5459e8857d796f4faf503 (127.10.238.193:40525)
I20260929 10:54:59.424110 16776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.193@KRBTEST.COM'} at 127.10.238.193:55325
WARNING: no policy specified for kudu/127.10.238.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab.
Entry for principal kudu/127.10.238.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab.
I20260929 10:54:59.446295 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.194:0
--local_ip_for_outbound_sockets=127.10.238.194
--webserver_interface=127.10.238.194
--webserver_port=0
--tserver_master_addrs=127.10.238.254:41619
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab
--principal=kudu/127.10.238.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:40833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:54:59.529476 16973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:54:59.529680 16973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:54:59.529702 16973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:54:59.530944 16973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:54:59.530985 16973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:54:59.531013 16973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.194
I20260929 10:54:59.532490 16973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:40833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab
--principal=kudu/127.10.238.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.238.194
--webserver_port=0
--tserver_master_addrs=127.10.238.254:41619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.194
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:54:59.532725 16973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:54:59.532953 16973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:54:59.537873 16978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:59.537878 16981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:59.538087 16973 server_base.cc:1034] running on GCE node
W20260929 10:54:59.538265 16979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:59.538491 16973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:54:59.538748 16973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:54:59.539920 16973 hybrid_clock.cc:648] HybridClock initialized: now 1790679299539920 us; error 43 us; skew 500 ppm
Sep 29 10:54:59 dist-test-slave-8nrt krb5kdc[16732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679299, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:54:59.542420 16973 init.cc:377] Logged in from keytab as kudu/127.10.238.194@KRBTEST.COM (short username kudu)
I20260929 10:54:59.542886 16973 webserver.cc:460] Webserver started at http://127.10.238.194:34797/ using document root <none> and password file <none>
I20260929 10:54:59.543125 16973 fs_manager.cc:359] Metadata directory not provided
I20260929 10:54:59.543191 16973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:54:59.543380 16973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:54:59.544412 16973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/ts-1/data/instance:
uuid: "8756253219384c88af3a404e47fca4af"
format_stamp: "Formatted at 2026-09-29 10:54:59 on dist-test-slave-8nrt"
server_key: "7c31561e105ebb621ab645556e373f27"
server_key_iv: "945eafc1a3b2c49c6432762084a3614d"
server_key_version: "encryptionkey@0"
I20260929 10:54:59.544782 16973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/ts-1/wal/instance:
uuid: "8756253219384c88af3a404e47fca4af"
format_stamp: "Formatted at 2026-09-29 10:54:59 on dist-test-slave-8nrt"
server_key: "7c31561e105ebb621ab645556e373f27"
server_key_iv: "945eafc1a3b2c49c6432762084a3614d"
server_key_version: "encryptionkey@0"
I20260929 10:54:59.546180 16973 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 10:54:59.546885 16988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:59.547062 16973 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:59.547160 16973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/ts-1/wal
uuid: "8756253219384c88af3a404e47fca4af"
format_stamp: "Formatted at 2026-09-29 10:54:59 on dist-test-slave-8nrt"
server_key: "7c31561e105ebb621ab645556e373f27"
server_key_iv: "945eafc1a3b2c49c6432762084a3614d"
server_key_version: "encryptionkey@0"
I20260929 10:54:59.547250 16973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:54:59.554389 16973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:54:59.556003 16973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:54:59.556156 16973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:54:59.556380 16973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:54:59.556658 16973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:54:59.556710 16973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:59.556747 16973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:54:59.556802 16973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:59.562620 16973 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.194:34417
I20260929 10:54:59.562673 17101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.194:34417 every 8 connection(s)
I20260929 10:54:59.562950 16973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
Sep 29 10:54:59 dist-test-slave-8nrt krb5kdc[16732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679299, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.194@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:54:59.570919 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 16973
I20260929 10:54:59.571048 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/ts-1/wal/instance
I20260929 10:54:59.571211 11195 external_mini_cluster.cc:1428] Setting key 561b7c343a749148309c6f7f441d150d
I20260929 10:54:59.571475 16817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:59.563595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:54969 (local address 127.10.238.254:41619)
0929 10:54:59.563670 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:59.563672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:59.563849 (+ 177us) server_negotiation.cc:407] Connection header received
0929 10:54:59.564712 (+ 863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:59.564713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:59.564723 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:59.564782 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:59.565378 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:59.565855 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:59.566758 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:59.566934 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:59.566982 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:59.568804 (+ 1822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:59.568805 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:59.568805 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:59.568812 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:59.570218 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:59.570561 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:59.570562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:59.570562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:59.570608 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:59.570865 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:59.570865 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:59.570865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:59.570954 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:59.571045 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:59.571315 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 10:54:59.571391 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 10:54:59.571655 17104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:59.563581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:41619 (local address 127.10.238.194:54969)
0929 10:54:59.563722 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:59.563763 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:59.564630 (+ 867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:59.564832 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:59.564835 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:59.565061 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:54:59.565246 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:59.565250 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:59.565929 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:59.565930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:59.566686 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:59.566688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:59.567001 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:59.567002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:59.567093 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:59.567497 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:54:59.567504 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:54:59.568726 (+ 1222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:54:59.570284 (+ 1558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:59.570290 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:59.570291 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:59.570500 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:59.570662 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:59.570663 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:59.570663 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:59.570801 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:59.571038 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:54:59.571045 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:54:59.571179 (+ 134us) client_negotiation.cc:769] Sending connection context
0929 10:54:59.571295 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 10:54:59.571431 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":78,"threads_started":1}
I20260929 10:54:59.572165 17102 heartbeater.cc:347] Connected to a master server at 127.10.238.254:41619
I20260929 10:54:59.572289 17102 heartbeater.cc:464] Registering TS with master...
I20260929 10:54:59.572456 17102 heartbeater.cc:511] Master 127.10.238.254:41619 requested a full tablet report, sending...
I20260929 10:54:59.572850 16776 ts_manager.cc:194] Registered new tserver with Master: 8756253219384c88af3a404e47fca4af (127.10.238.194:34417)
I20260929 10:54:59.573334 16776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.194@KRBTEST.COM'} at 127.10.238.194:54969
WARNING: no policy specified for kudu/127.10.238.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab.
Entry for principal kudu/127.10.238.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab.
I20260929 10:54:59.595873 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.195:0
--local_ip_for_outbound_sockets=127.10.238.195
--webserver_interface=127.10.238.195
--webserver_port=0
--tserver_master_addrs=127.10.238.254:41619
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab
--principal=kudu/127.10.238.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:40833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:54:59.675028 17107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:54:59.675230 17107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:54:59.675253 17107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:54:59.676316 17107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:54:59.676348 17107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:54:59.676393 17107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.195
I20260929 10:54:59.677773 17107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:40833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab
--principal=kudu/127.10.238.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.238.195
--webserver_port=0
--tserver_master_addrs=127.10.238.254:41619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.195
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:54:59.677940 17107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:54:59.678095 17107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:54:59.683012 17115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:59.683045 17107 server_base.cc:1034] running on GCE node
W20260929 10:54:59.683007 17112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:54:59.683007 17113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:54:59.683522 17107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:54:59.683789 17107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:54:59.684954 17107 hybrid_clock.cc:648] HybridClock initialized: now 1790679299684939 us; error 44 us; skew 500 ppm
Sep 29 10:54:59 dist-test-slave-8nrt krb5kdc[16732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679299, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:54:59.687479 17107 init.cc:377] Logged in from keytab as kudu/127.10.238.195@KRBTEST.COM (short username kudu)
I20260929 10:54:59.687865 17107 webserver.cc:460] Webserver started at http://127.10.238.195:40695/ using document root <none> and password file <none>
I20260929 10:54:59.688042 17107 fs_manager.cc:359] Metadata directory not provided
I20260929 10:54:59.688081 17107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:54:59.688189 17107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:54:59.689206 17107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/ts-2/data/instance:
uuid: "ce0ebde64eb6431b92603ecb168f9670"
format_stamp: "Formatted at 2026-09-29 10:54:59 on dist-test-slave-8nrt"
server_key: "8d0a3cc93a6ca316a071abc02d7640cb"
server_key_iv: "a6c3657deaa55d79d83a129825182a17"
server_key_version: "encryptionkey@0"
I20260929 10:54:59.689622 17107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/ts-2/wal/instance:
uuid: "ce0ebde64eb6431b92603ecb168f9670"
format_stamp: "Formatted at 2026-09-29 10:54:59 on dist-test-slave-8nrt"
server_key: "8d0a3cc93a6ca316a071abc02d7640cb"
server_key_iv: "a6c3657deaa55d79d83a129825182a17"
server_key_version: "encryptionkey@0"
I20260929 10:54:59.690928 17107 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:54:59.691630 17122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:59.691809 17107 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:54:59.691905 17107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/ts-2/wal
uuid: "ce0ebde64eb6431b92603ecb168f9670"
format_stamp: "Formatted at 2026-09-29 10:54:59 on dist-test-slave-8nrt"
server_key: "8d0a3cc93a6ca316a071abc02d7640cb"
server_key_iv: "a6c3657deaa55d79d83a129825182a17"
server_key_version: "encryptionkey@0"
I20260929 10:54:59.691980 17107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:54:59.724006 17107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:54:59.726491 17107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:54:59.726647 17107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:54:59.726881 17107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:54:59.727188 17107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:54:59.727244 17107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:59.727283 17107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:54:59.727313 17107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:54:59.733212 17107 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.195:45661
I20260929 10:54:59.733347 17235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.195:45661 every 8 connection(s)
I20260929 10:54:59.733611 17107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
Sep 29 10:54:59 dist-test-slave-8nrt krb5kdc[16732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679299, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.195@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:54:59.740417 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 17107
I20260929 10:54:59.740528 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/minicluster-data/ts-2/wal/instance
I20260929 10:54:59.740732 11195 external_mini_cluster.cc:1428] Setting key a72016e31046893c8a5b81ea075c6ae1
I20260929 10:54:59.742853 16817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:59.734328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:43747 (local address 127.10.238.254:41619)
0929 10:54:59.734369 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:59.734372 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:59.734574 (+ 202us) server_negotiation.cc:407] Connection header received
0929 10:54:59.735557 (+ 983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:59.735559 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:59.735573 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:59.735624 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:59.736233 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:59.736755 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:59.737823 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:59.738030 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:59.738115 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:59.740037 (+ 1922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:59.740038 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:59.740039 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:59.740047 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:59.741810 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:59.742167 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:59.742167 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:59.742168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:59.742217 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:59.742399 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:59.742400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:59.742400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:59.742495 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:59.742542 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:59.742735 (+ 193us) server_negotiation.cc:299] Negotiation successful
0929 10:54:59.742793 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 10:54:59.743009 17238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:59.734300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:41619 (local address 127.10.238.195:43747)
0929 10:54:59.734459 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 10:54:59.734493 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 10:54:59.735439 (+ 946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:54:59.735694 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:54:59.735703 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:54:59.735949 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:54:59.736132 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:59.736136 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:59.736860 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:59.736862 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:59.737726 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:54:59.737728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:54:59.738126 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:54:59.738127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:54:59.738253 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:59.738652 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:54:59.738660 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:54:59.739938 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:54:59.741891 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:59.741897 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:59.741899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:59.742105 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:59.742266 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:54:59.742266 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:54:59.742267 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:54:59.742352 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:54:59.742531 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:54:59.742533 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:54:59.742635 (+ 102us) client_negotiation.cc:769] Sending connection context
0929 10:54:59.742707 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 10:54:59.742814 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":87,"threads_started":1}
I20260929 10:54:59.743523 17236 heartbeater.cc:347] Connected to a master server at 127.10.238.254:41619
I20260929 10:54:59.743626 17236 heartbeater.cc:464] Registering TS with master...
I20260929 10:54:59.743796 17236 heartbeater.cc:511] Master 127.10.238.254:41619 requested a full tablet report, sending...
I20260929 10:54:59.744179 16776 ts_manager.cc:194] Registered new tserver with Master: ce0ebde64eb6431b92603ecb168f9670 (127.10.238.195:45661)
I20260929 10:54:59.744539 16776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.195@KRBTEST.COM'} at 127.10.238.195:43747
I20260929 10:54:59.752508 11195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/krb5kdc/test-admin.keytab.
Sep 29 10:54:59 dist-test-slave-8nrt krb5kdc[16732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:54:59.767335 11195 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260929 10:54:59.768823 17243 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:54:59.768132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54374 (local address 127.10.238.193:40525)
0929 10:54:59.768276 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:59.768286 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:59.768297 (+ 11us) server_negotiation.cc:407] Connection header received
0929 10:54:59.768322 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:59.768324 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:59.768336 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:59.768399 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:59.768722 (+ 323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54374: BlockingRecv error: recv got EOF from 127.0.0.1:54374 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
W20260929 10:54:59.768937 11195 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/krb5kdc/krb5cc' of type FILE
Sep 29 10:54:59 dist-test-slave-8nrt krb5kdc[16732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:54:59.770094 11195 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 10:54:59 dist-test-slave-8nrt krb5kdc[16732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:54:59.771008 11195 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 10:54:59 dist-test-slave-8nrt krb5kdc[16732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.238.193@KRBTEST.COM
I20260929 10:54:59.778813 17243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:59.771522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54386 (local address 127.10.238.193:40525)
0929 10:54:59.771572 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:59.771574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:59.771629 (+ 55us) server_negotiation.cc:407] Connection header received
0929 10:54:59.771686 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:59.771687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:59.771697 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:59.771740 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:59.772124 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:59.772870 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:59.773645 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:59.773960 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:59.774018 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:59.776170 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:59.776178 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:59.776178 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:59.776185 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:59.777825 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:59.778130 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:59.778133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:59.778133 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:59.778183 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:59.778340 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:59.778341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:59.778341 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:59.778459 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:59.778556 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:59.778618 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 10:54:59.778674 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260929 10:54:59.780741 17243 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:54:59.780120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54402 (local address 127.10.238.193:40525)
0929 10:54:59.780166 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:59.780167 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:59.780199 (+ 32us) server_negotiation.cc:407] Connection header received
0929 10:54:59.780251 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:59.780251 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:59.780260 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:59.780306 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:59.780691 (+ 385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54402: BlockingRecv error: recv got EOF from 127.0.0.1:54402 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260929 10:54:59.780838 11195 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/krb5kdc/krb5cc' of type FILE
Sep 29 10:54:59 dist-test-slave-8nrt krb5kdc[16732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:54:59.781842 11195 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 10:54:59 dist-test-slave-8nrt krb5kdc[16732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:54:59.782701 11195 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 10:54:59 dist-test-slave-8nrt krb5kdc[16732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.238.193@KRBTEST.COM
I20260929 10:54:59.789413 17243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:59.783235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54416 (local address 127.10.238.193:40525)
0929 10:54:59.783283 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:59.783285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:59.783295 (+ 10us) server_negotiation.cc:407] Connection header received
0929 10:54:59.783343 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:59.783344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:59.783352 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:59.783402 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:59.783723 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:59.784208 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:59.784990 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:59.785181 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:59.785244 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:59.787242 (+ 1998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:59.787243 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:59.787243 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:59.787249 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:59.788479 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:59.788767 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:59.788768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:59.788768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:59.788814 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:59.788958 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:59.788958 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:59.788959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:59.789034 (+ 75us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:59.789128 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:59.789232 (+ 104us) server_negotiation.cc:299] Negotiation successful
0929 10:54:59.789355 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20260929 10:54:59.790853 17243 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:54:59.790376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54426 (local address 127.10.238.193:40525)
0929 10:54:59.790413 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:59.790414 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:59.790422 (+ 8us) server_negotiation.cc:407] Connection header received
0929 10:54:59.790486 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:59.790487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:59.790494 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:59.790547 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:59.790793 (+ 246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54426: BlockingRecv error: recv got EOF from 127.0.0.1:54426 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260929 10:54:59.790884 11195 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790679274510869-11195-0/krb5kdc/krb5cc' of type FILE
Sep 29 10:54:59 dist-test-slave-8nrt krb5kdc[16732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:54:59.791785 11195 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 10:54:59 dist-test-slave-8nrt krb5kdc[16732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:54:59.792574 11195 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 10:54:59 dist-test-slave-8nrt krb5kdc[16732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.238.193@KRBTEST.COM
I20260929 10:54:59.799139 17243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:54:59.793009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54440 (local address 127.10.238.193:40525)
0929 10:54:59.793047 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 10:54:59.793049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:54:59.793078 (+ 29us) server_negotiation.cc:407] Connection header received
0929 10:54:59.793122 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:54:59.793123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:54:59.793129 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:54:59.793168 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:54:59.793535 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:59.794018 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:59.794698 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:54:59.794869 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:54:59.794916 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:54:59.796858 (+ 1942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:54:59.796859 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:54:59.796859 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:54:59.796865 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:54:59.798257 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:59.798545 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:59.798546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:59.798546 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:59.798592 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:54:59.798748 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:54:59.798748 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:54:59.798748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:54:59.798827 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:54:59.798887 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 10:54:59.798973 (+ 86us) server_negotiation.cc:299] Negotiation successful
0929 10:54:59.799021 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 10:54:59.799767 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 16839
I20260929 10:54:59.804299 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 16973
I20260929 10:54:59.810042 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 17107
I20260929 10:54:59.814283 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 16745
2026-09-29T10:54:59Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1878 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:54:59 dist-test-slave-8nrt krb5kdc[17256](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:54:59 dist-test-slave-8nrt krb5kdc[17256](info): set up 2 sockets
Sep 29 10:54:59 dist-test-slave-8nrt krb5kdc[17256](info): commencing operation
krb5kdc: starting...
W20260929 10:55:00.845546 11195 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Sep 29 10:55:00 dist-test-slave-8nrt krb5kdc[17256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:55:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:55:00Z Disabled control of system clock
WARNING: no policy specified for oryx/127.10.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.254.keytab.
Entry for principal oryx/127.10.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.254.keytab.
I20260929 10:55:00.962426 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:39293
--webserver_interface=127.10.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.254.keytab
--principal=oryx/127.10.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:46719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:39293
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:55:01.041152 17269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:55:01.041374 17269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:55:01.041395 17269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:55:01.042568 17269 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:55:01.042609 17269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:55:01.042623 17269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:55:01.042662 17269 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:55:01.042693 17269 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:55:01.044186 17269 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:46719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:39293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.254.keytab
--principal=oryx/127.10.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:39293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:55:01.044415 17269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:55:01.044625 17269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:55:01.049412 17275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:01.049419 17277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:55:01.049499 17269 server_base.cc:1034] running on GCE node
W20260929 10:55:01.049430 17274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:55:01.049836 17269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:55:01.050066 17269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:55:01.051199 17269 hybrid_clock.cc:648] HybridClock initialized: now 1790679301051181 us; error 41 us; skew 500 ppm
Sep 29 10:55:01 dist-test-slave-8nrt krb5kdc[17256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679301, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:55:01.053603 17269 init.cc:377] Logged in from keytab as oryx/127.10.238.254@KRBTEST.COM (short username oryx)
I20260929 10:55:01.054132 17269 webserver.cc:460] Webserver started at http://127.10.238.254:37231/ using document root <none> and password file <none>
I20260929 10:55:01.054353 17269 fs_manager.cc:359] Metadata directory not provided
I20260929 10:55:01.054399 17269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:55:01.054556 17269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:55:01.055646 17269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-0/data/instance:
uuid: "82fdb61571ad46209f5fe7dca6fae4bf"
format_stamp: "Formatted at 2026-09-29 10:55:01 on dist-test-slave-8nrt"
server_key: "7a58b383bee7b3270880ec15ce837220"
server_key_iv: "d55172103d532413ef3d5e590ed8a046"
server_key_version: "encryptionkey@0"
I20260929 10:55:01.056039 17269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-0/wal/instance:
uuid: "82fdb61571ad46209f5fe7dca6fae4bf"
format_stamp: "Formatted at 2026-09-29 10:55:01 on dist-test-slave-8nrt"
server_key: "7a58b383bee7b3270880ec15ce837220"
server_key_iv: "d55172103d532413ef3d5e590ed8a046"
server_key_version: "encryptionkey@0"
I20260929 10:55:01.057497 17269 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 10:55:01.058202 17284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:01.058383 17269 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:55:01.058456 17269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "82fdb61571ad46209f5fe7dca6fae4bf"
format_stamp: "Formatted at 2026-09-29 10:55:01 on dist-test-slave-8nrt"
server_key: "7a58b383bee7b3270880ec15ce837220"
server_key_iv: "d55172103d532413ef3d5e590ed8a046"
server_key_version: "encryptionkey@0"
I20260929 10:55:01.058514 17269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:55:01.080035 17269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:55:01.082027 17269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:55:01.082160 17269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:55:01.086580 17269 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.254:39293
I20260929 10:55:01.086606 17336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.254:39293 every 8 connection(s)
I20260929 10:55:01.087010 17269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:55:01.088095 17337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:55:01.090345 17337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf: Bootstrap starting.
I20260929 10:55:01.090880 17337 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf: Neither blocks nor log segments found. Creating new log.
I20260929 10:55:01.091104 17337 log.cc:824] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf: Log is configured to *not* fsync() on all Append() calls
I20260929 10:55:01.091753 17337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf: No bootstrap required, opened a new log
I20260929 10:55:01.093302 17337 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } }
I20260929 10:55:01.093456 17337 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:55:01.093478 17337 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82fdb61571ad46209f5fe7dca6fae4bf, State: Initialized, Role: FOLLOWER
I20260929 10:55:01.093576 17337 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } }
I20260929 10:55:01.093659 17337 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:55:01.093696 17337 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:55:01.093747 17337 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:55:01.094527 17337 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } }
I20260929 10:55:01.094662 17337 leader_election.cc:302] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82fdb61571ad46209f5fe7dca6fae4bf; no voters:
I20260929 10:55:01.094846 17337 leader_election.cc:288] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:55:01.094885 17340 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:55:01.095073 17340 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf [term 1 LEADER]: Becoming Leader. State: Replica: 82fdb61571ad46209f5fe7dca6fae4bf, State: Running, Role: LEADER
I20260929 10:55:01.095106 17337 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:55:01.095223 17340 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } }
I20260929 10:55:01.095549 17341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } } }
I20260929 10:55:01.095561 17342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82fdb61571ad46209f5fe7dca6fae4bf. Latest consensus state: current_term: 1 leader_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } } }
I20260929 10:55:01.095726 17342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf [sys.catalog]: This master's current role is: LEADER
I20260929 10:55:01.095717 17341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf [sys.catalog]: This master's current role is: LEADER
I20260929 10:55:01.096153 17348 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:55:01.096675 17348 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:55:01.096810 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 17269
I20260929 10:55:01.096936 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-0/wal/instance
I20260929 10:55:01.097144 11195 external_mini_cluster.cc:1428] Setting key 507299a994cd990d22aac63fe4a9580a
I20260929 10:55:01.102209 17348 catalog_manager.cc:1347] Generated new cluster ID: 8ee41e681375487bbdaa5339ed2563f0
I20260929 10:55:01.102275 17348 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 10:55:01 dist-test-slave-8nrt krb5kdc[17256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM
I20260929 10:55:01.109305 17348 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:55:01.109942 17348 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:55:01.110625 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:01.097579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54772 (local address 127.10.238.254:39293)
0929 10:55:01.101405 (+ 3826us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:01.101414 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:01.101450 (+ 36us) server_negotiation.cc:407] Connection header received
0929 10:55:01.101563 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:01.101570 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:01.101634 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:01.101779 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:01.102532 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:01.103743 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:01.104515 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:01.104923 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:01.105040 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:01.107269 (+ 2229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:01.107272 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:01.107273 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:01.107291 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:01.109033 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:01.109478 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:01.109481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:01.109483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:01.109572 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:01.109784 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:01.109785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:01.109785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:01.110021 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:01.110181 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:01.110247 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 10:55:01.110410 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3780,"thread_start_us":106,"threads_started":1}
I20260929 10:55:01.120692 17348 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf: Generated new TSK 0
I20260929 10:55:01.120889 17348 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.10.238.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.238.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.238.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.193.keytab.
Entry for principal oryx/127.10.238.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.193.keytab.
I20260929 10:55:01.152107 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.193:0
--local_ip_for_outbound_sockets=127.10.238.193
--webserver_interface=127.10.238.193
--webserver_port=0
--tserver_master_addrs=127.10.238.254:39293
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.193.keytab
--principal=oryx/127.10.238.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:46719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:55:01.249053 17363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:55:01.249245 17363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:55:01.249284 17363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:55:01.250455 17363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:55:01.250499 17363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:55:01.250524 17363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.193
I20260929 10:55:01.251969 17363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:46719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.193.keytab
--principal=oryx/127.10.238.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.238.193
--webserver_port=0
--tserver_master_addrs=127.10.238.254:39293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.193
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:55:01.252218 17363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:55:01.252447 17363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:55:01.257431 17369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:01.257436 17368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:01.257490 17371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:55:01.257576 17363 server_base.cc:1034] running on GCE node
I20260929 10:55:01.257763 17363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:55:01.258049 17363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:55:01.259213 17363 hybrid_clock.cc:648] HybridClock initialized: now 1790679301259193 us; error 36 us; skew 500 ppm
Sep 29 10:55:01 dist-test-slave-8nrt krb5kdc[17256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679301, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.238.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:55:01.261786 17363 init.cc:377] Logged in from keytab as oryx/127.10.238.193@KRBTEST.COM (short username oryx)
I20260929 10:55:01.262331 17363 webserver.cc:460] Webserver started at http://127.10.238.193:43063/ using document root <none> and password file <none>
I20260929 10:55:01.262578 17363 fs_manager.cc:359] Metadata directory not provided
I20260929 10:55:01.262645 17363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:55:01.262792 17363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:55:01.263849 17363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-0/data/instance:
uuid: "2b18011dc9b24cf587c3bb2237afc7e6"
format_stamp: "Formatted at 2026-09-29 10:55:01 on dist-test-slave-8nrt"
server_key: "b2057d7f0337f4cab3433171432cb795"
server_key_iv: "bdc6f0ba15c1ecde67411c421764c0fe"
server_key_version: "encryptionkey@0"
I20260929 10:55:01.264220 17363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-0/wal/instance:
uuid: "2b18011dc9b24cf587c3bb2237afc7e6"
format_stamp: "Formatted at 2026-09-29 10:55:01 on dist-test-slave-8nrt"
server_key: "b2057d7f0337f4cab3433171432cb795"
server_key_iv: "bdc6f0ba15c1ecde67411c421764c0fe"
server_key_version: "encryptionkey@0"
I20260929 10:55:01.265571 17363 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 10:55:01.266281 17378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:01.266456 17363 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 10:55:01.266554 17363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-0/wal
uuid: "2b18011dc9b24cf587c3bb2237afc7e6"
format_stamp: "Formatted at 2026-09-29 10:55:01 on dist-test-slave-8nrt"
server_key: "b2057d7f0337f4cab3433171432cb795"
server_key_iv: "bdc6f0ba15c1ecde67411c421764c0fe"
server_key_version: "encryptionkey@0"
I20260929 10:55:01.266640 17363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:55:01.283701 17363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:55:01.285679 17363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:55:01.285848 17363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:55:01.286090 17363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:55:01.286415 17363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:55:01.286469 17363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:01.286504 17363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:55:01.286531 17363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:01.292563 17363 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.193:42961
I20260929 10:55:01.292609 17491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.193:42961 every 8 connection(s)
I20260929 10:55:01.292935 17363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
I20260929 10:55:01.297652 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 17363
I20260929 10:55:01.297778 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-0/wal/instance
I20260929 10:55:01.297987 11195 external_mini_cluster.cc:1428] Setting key 982f5755291ddee099691b5b69069dbf
Sep 29 10:55:01 dist-test-slave-8nrt krb5kdc[17256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679301, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.238.193@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM
I20260929 10:55:01.302780 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:01.293673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:55929 (local address 127.10.238.254:39293)
0929 10:55:01.293766 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:01.293769 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:01.293971 (+ 202us) server_negotiation.cc:407] Connection header received
0929 10:55:01.294479 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:01.294481 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:01.294492 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:01.294548 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:01.295525 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:01.296107 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:01.297228 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:01.297465 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:01.297542 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:01.299802 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:01.299804 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:01.299804 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:01.299813 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:01.301248 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:01.301787 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:01.301788 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:01.301789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:01.301839 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:01.302132 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:01.302132 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:01.302132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:01.302236 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:01.302316 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:01.302636 (+ 320us) server_negotiation.cc:299] Negotiation successful
0929 10:55:01.302716 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 10:55:01.303084 17494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:01.293643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39293 (local address 127.10.238.193:55929)
0929 10:55:01.293842 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:01.293878 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:01.294387 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:01.294627 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:01.294630 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:01.295254 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:01.295448 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:01.295453 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:01.296214 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:01.296215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:01.297148 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:01.297151 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:01.297559 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:01.297560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:01.297663 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:01.298247 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:01.298257 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:01.299710 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:55:01.301373 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:01.301380 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:01.301382 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:01.301710 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:01.301901 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:01.301902 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:01.301903 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:01.302066 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:01.302307 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:55:01.302315 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:55:01.302492 (+ 177us) client_negotiation.cc:769] Sending connection context
0929 10:55:01.302617 (+ 125us) client_negotiation.cc:240] Negotiation successful
0929 10:55:01.302800 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":120,"threads_started":1}
I20260929 10:55:01.303653 17492 heartbeater.cc:347] Connected to a master server at 127.10.238.254:39293
I20260929 10:55:01.303766 17492 heartbeater.cc:464] Registering TS with master...
I20260929 10:55:01.303978 17492 heartbeater.cc:511] Master 127.10.238.254:39293 requested a full tablet report, sending...
I20260929 10:55:01.304533 17301 ts_manager.cc:194] Registered new tserver with Master: 2b18011dc9b24cf587c3bb2237afc7e6 (127.10.238.193:42961)
I20260929 10:55:01.305106 17301 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.238.193@KRBTEST.COM'} at 127.10.238.193:55929
WARNING: no policy specified for oryx/127.10.238.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.238.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.238.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.194.keytab.
Entry for principal oryx/127.10.238.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.194.keytab.
I20260929 10:55:01.324213 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.194:0
--local_ip_for_outbound_sockets=127.10.238.194
--webserver_interface=127.10.238.194
--webserver_port=0
--tserver_master_addrs=127.10.238.254:39293
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.194.keytab
--principal=oryx/127.10.238.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:46719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:55:01.403798 17497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:55:01.403988 17497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:55:01.404011 17497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:55:01.405154 17497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:55:01.405193 17497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:55:01.405220 17497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.194
I20260929 10:55:01.406648 17497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:46719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.194.keytab
--principal=oryx/127.10.238.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.238.194
--webserver_port=0
--tserver_master_addrs=127.10.238.254:39293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17497
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.194
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:55:01.406828 17497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:55:01.407055 17497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:55:01.412081 17503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:01.412182 17505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:55:01.412096 17497 server_base.cc:1034] running on GCE node
W20260929 10:55:01.412245 17502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:55:01.412539 17497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:55:01.412787 17497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:55:01.413946 17497 hybrid_clock.cc:648] HybridClock initialized: now 1790679301413925 us; error 37 us; skew 500 ppm
Sep 29 10:55:01 dist-test-slave-8nrt krb5kdc[17256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679301, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.238.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:55:01.416342 17497 init.cc:377] Logged in from keytab as oryx/127.10.238.194@KRBTEST.COM (short username oryx)
I20260929 10:55:01.416783 17497 webserver.cc:460] Webserver started at http://127.10.238.194:41693/ using document root <none> and password file <none>
I20260929 10:55:01.417011 17497 fs_manager.cc:359] Metadata directory not provided
I20260929 10:55:01.417078 17497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:55:01.417229 17497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:55:01.418275 17497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-1/data/instance:
uuid: "bb2eb90f68da49f7b9fb25af83502e1f"
format_stamp: "Formatted at 2026-09-29 10:55:01 on dist-test-slave-8nrt"
server_key: "e6e721703a907ea8d01f2a9ef03bac35"
server_key_iv: "9977ed11a932968af2e929a98e8c5e6e"
server_key_version: "encryptionkey@0"
I20260929 10:55:01.418653 17497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-1/wal/instance:
uuid: "bb2eb90f68da49f7b9fb25af83502e1f"
format_stamp: "Formatted at 2026-09-29 10:55:01 on dist-test-slave-8nrt"
server_key: "e6e721703a907ea8d01f2a9ef03bac35"
server_key_iv: "9977ed11a932968af2e929a98e8c5e6e"
server_key_version: "encryptionkey@0"
I20260929 10:55:01.419894 17497 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 10:55:01.420562 17512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:01.420790 17497 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 10:55:01.420897 17497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-1/wal
uuid: "bb2eb90f68da49f7b9fb25af83502e1f"
format_stamp: "Formatted at 2026-09-29 10:55:01 on dist-test-slave-8nrt"
server_key: "e6e721703a907ea8d01f2a9ef03bac35"
server_key_iv: "9977ed11a932968af2e929a98e8c5e6e"
server_key_version: "encryptionkey@0"
I20260929 10:55:01.420975 17497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:55:01.434729 17497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:55:01.436378 17497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:55:01.436545 17497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:55:01.436755 17497 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:55:01.437045 17497 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:55:01.437103 17497 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:01.437143 17497 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:55:01.437171 17497 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:01.442925 17497 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.194:40983
I20260929 10:55:01.442967 17625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.194:40983 every 8 connection(s)
I20260929 10:55:01.443276 17497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
I20260929 10:55:01.449033 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 17497
I20260929 10:55:01.449147 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-1/wal/instance
Sep 29 10:55:01 dist-test-slave-8nrt krb5kdc[17256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679301, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.238.194@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM
I20260929 10:55:01.449414 11195 external_mini_cluster.cc:1428] Setting key cccd0b5a10ba5482fa3500b4da11861f
I20260929 10:55:01.452888 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:01.443939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:34891 (local address 127.10.238.254:39293)
0929 10:55:01.443981 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:01.443983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:01.444176 (+ 193us) server_negotiation.cc:407] Connection header received
0929 10:55:01.444734 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:01.444735 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:01.444749 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:01.444804 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:01.445789 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:01.446382 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:01.447362 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:01.447562 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:01.447637 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:01.449884 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:01.449885 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:01.449885 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:01.449891 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:01.451526 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:01.451949 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:01.451950 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:01.451951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:01.452016 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:01.452268 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:01.452269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:01.452269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:01.452382 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:01.452458 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:01.452716 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 10:55:01.452804 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 10:55:01.453060 17628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:01.443913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39293 (local address 127.10.238.194:34891)
0929 10:55:01.444057 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:01.444092 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:01.444611 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:01.444867 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:01.444871 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:01.445482 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:01.445664 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:01.445669 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:01.446479 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:01.446480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:01.447268 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:01.447270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:01.447643 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:01.447644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:01.447770 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:01.448275 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:01.448283 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:01.449770 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:55:01.451604 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:01.451611 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:01.451612 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:01.451867 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:01.452085 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:01.452086 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:01.452086 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:01.452192 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:01.452450 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:55:01.452457 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:55:01.452585 (+ 128us) client_negotiation.cc:769] Sending connection context
0929 10:55:01.452684 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 10:55:01.452835 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":76,"threads_started":1}
I20260929 10:55:01.453516 17626 heartbeater.cc:347] Connected to a master server at 127.10.238.254:39293
I20260929 10:55:01.453614 17626 heartbeater.cc:464] Registering TS with master...
I20260929 10:55:01.453796 17626 heartbeater.cc:511] Master 127.10.238.254:39293 requested a full tablet report, sending...
I20260929 10:55:01.454185 17301 ts_manager.cc:194] Registered new tserver with Master: bb2eb90f68da49f7b9fb25af83502e1f (127.10.238.194:40983)
I20260929 10:55:01.454566 17301 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.238.194@KRBTEST.COM'} at 127.10.238.194:34891
WARNING: no policy specified for oryx/127.10.238.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.238.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.238.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.195.keytab.
Entry for principal oryx/127.10.238.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.195.keytab.
I20260929 10:55:01.475765 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.195:0
--local_ip_for_outbound_sockets=127.10.238.195
--webserver_interface=127.10.238.195
--webserver_port=0
--tserver_master_addrs=127.10.238.254:39293
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.195.keytab
--principal=oryx/127.10.238.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:46719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:55:01.553933 17631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:55:01.554112 17631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:55:01.554136 17631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:55:01.555197 17631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:55:01.555227 17631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:55:01.555253 17631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.195
I20260929 10:55:01.556593 17631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:46719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.195.keytab
--principal=oryx/127.10.238.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.238.195
--webserver_port=0
--tserver_master_addrs=127.10.238.254:39293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.195
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:55:01.556770 17631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:55:01.556937 17631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:55:01.562049 17637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:01.562090 17636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:01.562180 17639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:55:01.562173 17631 server_base.cc:1034] running on GCE node
I20260929 10:55:01.562525 17631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:55:01.562757 17631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:55:01.563930 17631 hybrid_clock.cc:648] HybridClock initialized: now 1790679301563915 us; error 31 us; skew 500 ppm
Sep 29 10:55:01 dist-test-slave-8nrt krb5kdc[17256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679301, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.238.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:55:01.571033 17631 init.cc:377] Logged in from keytab as oryx/127.10.238.195@KRBTEST.COM (short username oryx)
I20260929 10:55:01.571491 17631 webserver.cc:460] Webserver started at http://127.10.238.195:40747/ using document root <none> and password file <none>
I20260929 10:55:01.571743 17631 fs_manager.cc:359] Metadata directory not provided
I20260929 10:55:01.571811 17631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:55:01.571956 17631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:55:01.572957 17631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-2/data/instance:
uuid: "99c69ddb2aee461092fbe82090413694"
format_stamp: "Formatted at 2026-09-29 10:55:01 on dist-test-slave-8nrt"
server_key: "b9a5e4a582e51d8608b150431979766e"
server_key_iv: "d5d7c6bb6917d57d997a553b30387173"
server_key_version: "encryptionkey@0"
I20260929 10:55:01.573370 17631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-2/wal/instance:
uuid: "99c69ddb2aee461092fbe82090413694"
format_stamp: "Formatted at 2026-09-29 10:55:01 on dist-test-slave-8nrt"
server_key: "b9a5e4a582e51d8608b150431979766e"
server_key_iv: "d5d7c6bb6917d57d997a553b30387173"
server_key_version: "encryptionkey@0"
I20260929 10:55:01.574635 17631 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 10:55:01.575420 17646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:01.575616 17631 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 10:55:01.575733 17631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-2/wal
uuid: "99c69ddb2aee461092fbe82090413694"
format_stamp: "Formatted at 2026-09-29 10:55:01 on dist-test-slave-8nrt"
server_key: "b9a5e4a582e51d8608b150431979766e"
server_key_iv: "d5d7c6bb6917d57d997a553b30387173"
server_key_version: "encryptionkey@0"
I20260929 10:55:01.575809 17631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:55:01.592406 17631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:55:01.594265 17631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:55:01.594420 17631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:55:01.594640 17631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:55:01.594995 17631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:55:01.595046 17631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:01.595085 17631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:55:01.595111 17631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:01.601334 17631 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.195:45979
I20260929 10:55:01.601363 17759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.195:45979 every 8 connection(s)
I20260929 10:55:01.601675 17631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
Sep 29 10:55:01 dist-test-slave-8nrt krb5kdc[17256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679301, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.238.195@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM
I20260929 10:55:01.610281 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:01.602453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:33523 (local address 127.10.238.254:39293)
0929 10:55:01.602547 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:01.602554 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:01.602752 (+ 198us) server_negotiation.cc:407] Connection header received
0929 10:55:01.603217 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:01.603218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:01.603234 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:01.603290 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:01.604094 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:01.604571 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:01.605551 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:01.605734 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:01.605786 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:01.607602 (+ 1816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:01.607603 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:01.607603 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:01.607610 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:01.608944 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:01.609499 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:01.609500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:01.609500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:01.609555 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:01.609773 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:01.609773 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:01.609774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:01.609872 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:01.609922 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:01.610145 (+ 223us) server_negotiation.cc:299] Negotiation successful
0929 10:55:01.610209 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 10:55:01.610329 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 17631
I20260929 10:55:01.610415 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-2/wal/instance
I20260929 10:55:01.610579 17762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:01.602424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39293 (local address 127.10.238.195:33523)
0929 10:55:01.602610 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:01.602650 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:01.603141 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:01.603337 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:01.603340 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:01.603852 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:01.604026 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:01.604030 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:01.604647 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:01.604648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:01.605486 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:01.605488 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:01.605804 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:01.605805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:01.605897 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:01.606303 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:01.606311 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:01.607523 (+ 1212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:55:01.609109 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:01.609115 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:01.609117 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:01.609424 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:01.609621 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:01.609621 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:01.609622 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:01.609718 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:01.609927 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:55:01.609932 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:55:01.610038 (+ 106us) client_negotiation.cc:769] Sending connection context
0929 10:55:01.610118 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 10:55:01.610305 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":99,"threads_started":1}
I20260929 10:55:01.610634 11195 external_mini_cluster.cc:1428] Setting key 938fce8fa8cf37ac229b7a6933535c44
I20260929 10:55:01.611198 17760 heartbeater.cc:347] Connected to a master server at 127.10.238.254:39293
I20260929 10:55:01.611346 17760 heartbeater.cc:464] Registering TS with master...
I20260929 10:55:01.611537 17760 heartbeater.cc:511] Master 127.10.238.254:39293 requested a full tablet report, sending...
I20260929 10:55:01.611940 17300 ts_manager.cc:194] Registered new tserver with Master: 99c69ddb2aee461092fbe82090413694 (127.10.238.195:45979)
I20260929 10:55:01.612319 17300 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.238.195@KRBTEST.COM'} at 127.10.238.195:33523
I20260929 10:55:01.622267 11195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.10.238.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.238.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.238.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.253.keytab.
Entry for principal oryx/127.10.238.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.253.keytab.
I20260929 10:55:01.647939 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.253:45709
--webserver_interface=127.10.238.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.253.keytab
--principal=oryx/127.10.238.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:46719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:39293,127.10.238.253:45709
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:55:01.736828 17765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:55:01.737023 17765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:55:01.737046 17765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:55:01.738219 17765 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:55:01.738261 17765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:55:01.738276 17765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:55:01.738328 17765 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:55:01.738346 17765 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:55:01.739758 17765 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:46719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:39293,127.10.238.253:45709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.253.keytab
--principal=oryx/127.10.238.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.253:45709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.238.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
W20260929 10:55:01.739879 17765 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.10.238.254:39293,127.10.238.253:45709'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 10:55:01.740017 17765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:55:01.740257 17765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:55:01.744930 17770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:01.745066 17773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:01.744935 17771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:55:01.745108 17765 server_base.cc:1034] running on GCE node
I20260929 10:55:01.745448 17765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:55:01.745675 17765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:55:01.746846 17765 hybrid_clock.cc:648] HybridClock initialized: now 1790679301746837 us; error 45 us; skew 500 ppm
Sep 29 10:55:01 dist-test-slave-8nrt krb5kdc[17256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679301, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.238.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:55:01.749176 17765 init.cc:377] Logged in from keytab as oryx/127.10.238.253@KRBTEST.COM (short username oryx)
I20260929 10:55:01.749627 17765 webserver.cc:460] Webserver started at http://127.10.238.253:43463/ using document root <none> and password file <none>
I20260929 10:55:01.749840 17765 fs_manager.cc:359] Metadata directory not provided
I20260929 10:55:01.749904 17765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:55:01.750046 17765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:55:01.751055 17765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-1/data/instance:
uuid: "59092f98066b4395b785a2698bf678c7"
format_stamp: "Formatted at 2026-09-29 10:55:01 on dist-test-slave-8nrt"
server_key: "05ddcb71b1da9d853e47c87c39cf63d2"
server_key_iv: "7908daccdafec69643aecd3e0179ceca"
server_key_version: "encryptionkey@0"
I20260929 10:55:01.751411 17765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-1/wal/instance:
uuid: "59092f98066b4395b785a2698bf678c7"
format_stamp: "Formatted at 2026-09-29 10:55:01 on dist-test-slave-8nrt"
server_key: "05ddcb71b1da9d853e47c87c39cf63d2"
server_key_iv: "7908daccdafec69643aecd3e0179ceca"
server_key_version: "encryptionkey@0"
I20260929 10:55:01.752671 17765 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 10:55:01.753422 17780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:01.753594 17765 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:55:01.753690 17765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-1/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-1/wal
uuid: "59092f98066b4395b785a2698bf678c7"
format_stamp: "Formatted at 2026-09-29 10:55:01 on dist-test-slave-8nrt"
server_key: "05ddcb71b1da9d853e47c87c39cf63d2"
server_key_iv: "7908daccdafec69643aecd3e0179ceca"
server_key_version: "encryptionkey@0"
I20260929 10:55:01.753798 17765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:55:01.762717 17765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:55:01.764468 17765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:55:01.764613 17765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:55:01.768517 17765 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.253:45709
I20260929 10:55:01.768559 17832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.253:45709 every 8 connection(s)
I20260929 10:55:01.768896 17765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-1/data/info.pb
I20260929 10:55:01.769871 17833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:55:01.771325 17833 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } has no permanent_uuid. Determining permanent_uuid...
I20260929 10:55:01.773276 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 17765
Sep 29 10:55:01 dist-test-slave-8nrt krb5kdc[17256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679301, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.238.253@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM
Sep 29 10:55:01 dist-test-slave-8nrt krb5kdc[17256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.238.253@KRBTEST.COM
I20260929 10:55:01.781363 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:01.772007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54780 (local address 127.10.238.254:39293)
0929 10:55:01.772067 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:01.772069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:01.772253 (+ 184us) server_negotiation.cc:407] Connection header received
0929 10:55:01.772682 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:01.772683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:01.772692 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:01.772753 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:01.773473 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:01.774253 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:01.775461 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:01.775768 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:01.775835 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:01.778208 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:01.778210 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:01.778210 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:01.778218 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:01.779844 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:01.780298 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:01.780299 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:01.780299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:01.780376 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:01.780625 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:01.780626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:01.780627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:01.780824 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:01.780901 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:01.781129 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 10:55:01.781215 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 10:55:01.781517 17835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:01.771973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39293 (local address 127.0.0.1:54780)
0929 10:55:01.772129 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:01.772169 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:01.772604 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:01.772811 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:01.772815 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:01.773042 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:01.773372 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:01.773380 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:01.774363 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:01.774364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:01.775377 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:01.775380 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:01.775850 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:01.775850 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:01.775979 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:01.776488 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:01.776497 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:01.778116 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:55:01.779927 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:01.779934 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:01.779935 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:01.780210 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:01.780456 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:01.780456 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:01.780457 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:01.780555 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:01.780896 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:55:01.780900 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:55:01.781004 (+ 104us) client_negotiation.cc:769] Sending connection context
0929 10:55:01.781081 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 10:55:01.781227 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
I20260929 10:55:01.782171 17833 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45709 } has no permanent_uuid. Determining permanent_uuid...
I20260929 10:55:01.782711 17837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:01.773798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47928 (local address 127.10.238.253:45709)
0929 10:55:01.773967 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:01.773971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:01.774023 (+ 52us) server_negotiation.cc:407] Connection header received
0929 10:55:01.774053 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:01.774055 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:01.774074 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:01.774148 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:01.774874 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:01.775672 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:01.776644 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:01.776927 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:01.776980 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:01.779441 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:01.779448 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:01.779449 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:01.779456 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:01.781233 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:01.781628 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:01.781631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:01.781631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:01.781691 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:01.781992 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:01.781992 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:01.781993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:01.782143 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:01.782278 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:01.782373 (+ 95us) server_negotiation.cc:299] Negotiation successful
0929 10:55:01.782585 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":109,"threads_started":1}
Sep 29 10:55:01 dist-test-slave-8nrt krb5kdc[17256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679301, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.238.253@KRBTEST.COM for oryx/127.10.238.253@KRBTEST.COM
I20260929 10:55:01.788621 17835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:01.782409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.253:45709 (local address 127.0.0.1:47940)
0929 10:55:01.782450 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:01.782458 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:01.782489 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:01.782702 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:01.782703 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:01.782871 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:01.782921 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:01.782923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:01.783656 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:01.783657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:01.784134 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:01.784135 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:01.784483 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:01.784484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:01.784536 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:01.784849 (+ 313us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:01.784853 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:01.785921 (+ 1068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:55:01.787890 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:01.787891 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:01.787891 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:01.788030 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:01.788193 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:01.788193 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:01.788193 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:01.788247 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:01.788421 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:55:01.788421 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:55:01.788453 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 10:55:01.788501 (+ 48us) client_negotiation.cc:240] Negotiation successful
0929 10:55:01.788567 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260929 10:55:01.788785 17838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:01.782458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47940 (local address 127.10.238.253:45709)
0929 10:55:01.782580 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:01.782582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:01.782589 (+ 7us) server_negotiation.cc:407] Connection header received
0929 10:55:01.782607 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:01.782608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:01.782617 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:01.782685 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:01.782983 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:01.783567 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:01.784221 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:01.784421 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:01.784465 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:01.785991 (+ 1526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:01.785992 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:01.785993 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:01.785998 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:01.787822 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:01.788089 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:01.788090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:01.788090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:01.788138 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:01.788291 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:01.788292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:01.788292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:01.788372 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:01.788414 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:01.788514 (+ 100us) server_negotiation.cc:299] Negotiation successful
0929 10:55:01.788609 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":103,"thread_start_us":78,"threads_started":1}
I20260929 10:55:01.790050 17833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7: Bootstrap starting.
I20260929 10:55:01.790652 17833 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7: Neither blocks nor log segments found. Creating new log.
I20260929 10:55:01.790931 17833 log.cc:824] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7: Log is configured to *not* fsync() on all Append() calls
I20260929 10:55:01.791611 17833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7: No bootstrap required, opened a new log
I20260929 10:55:01.792969 17833 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } } peers { permanent_uuid: "59092f98066b4395b785a2698bf678c7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45709 } }
I20260929 10:55:01.793082 17833 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:55:01.793129 17833 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59092f98066b4395b785a2698bf678c7, State: Initialized, Role: FOLLOWER
I20260929 10:55:01.793293 17833 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } } peers { permanent_uuid: "59092f98066b4395b785a2698bf678c7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45709 } }
I20260929 10:55:01.793553 17840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } } peers { permanent_uuid: "59092f98066b4395b785a2698bf678c7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45709 } } }
I20260929 10:55:01.793605 17833 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:55:01.793671 17840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 10:55:01.795107 17851 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:55:01.795179 17851 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:55:01.796705 17765 master_runner.cc:428] Detected that this master 59092f98066b4395b785a2698bf678c7 is joining an existing cluster
I20260929 10:55:01.796741 17765 master_runner.cc:432] Initiating AddMaster RPC to add 127.10.238.253:45709
I20260929 10:55:01.804693 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:01.797783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54786 (local address 127.10.238.254:39293)
0929 10:55:01.797828 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:01.797830 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:01.797969 (+ 139us) server_negotiation.cc:407] Connection header received
0929 10:55:01.798031 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:01.798032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:01.798040 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:01.798110 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:01.798499 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:01.799297 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:01.800348 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:01.800523 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:01.800603 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:01.801661 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:01.801662 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:01.801663 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:01.801670 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:01.803436 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:01.803838 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:01.803840 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:01.803840 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:01.803896 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:01.804088 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:01.804089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:01.804089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:01.804206 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:01.804247 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:01.804489 (+ 242us) server_negotiation.cc:299] Negotiation successful
0929 10:55:01.804569 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 10:55:01.805078 17858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:01.797718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39293 (local address 127.0.0.1:54786)
0929 10:55:01.797914 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:01.797924 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:01.797982 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:01.798114 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:01.798115 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:01.798343 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:01.798403 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:01.798405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:01.799384 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:01.799385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:01.800266 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:01.800269 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:01.800626 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:01.800628 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:01.800703 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:01.801142 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:01.801147 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:01.801570 (+ 423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:55:01.803538 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:01.803539 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:01.803540 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:01.803753 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:01.803967 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:01.803968 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:01.803968 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:01.804023 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:01.804293 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:55:01.804294 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:55:01.804333 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 10:55:01.804432 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 10:55:01.804548 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":103,"threads_started":1}
I20260929 10:55:01.805474 17838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:01.797916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47956 (local address 127.10.238.253:45709)
0929 10:55:01.797945 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:01.797947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:01.798107 (+ 160us) server_negotiation.cc:407] Connection header received
0929 10:55:01.798168 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:01.798169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:01.798178 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:01.798227 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:01.798613 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:01.799303 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:01.801303 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:01.801606 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:01.801663 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:01.802821 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:01.802822 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:01.802822 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:01.802828 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:01.804242 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:01.804689 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:01.804689 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:01.804690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:01.804737 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:01.804904 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:01.804905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:01.804905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:01.805011 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:01.805089 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:01.805234 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 10:55:01.805381 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260929 10:55:01.805799 17857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:01.797759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.253:45709 (local address 127.0.0.1:47956)
0929 10:55:01.798062 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:01.798070 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:01.798123 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:01.798233 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:01.798234 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:01.798475 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:01.798535 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:01.798537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:01.800366 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:01.800367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:01.801188 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:01.801190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:01.801703 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:01.801703 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:01.801756 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:01.802225 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:01.802230 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:01.802548 (+ 318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:55:01.804347 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:01.804348 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:01.804349 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:01.804612 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:01.804807 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:01.804808 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:01.804808 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:01.804857 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:01.805082 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:55:01.805083 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:55:01.805120 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 10:55:01.805600 (+ 480us) client_negotiation.cc:240] Negotiation successful
0929 10:55:01.805690 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":83,"threads_started":1}
Sep 29 10:55:01 dist-test-slave-8nrt krb5kdc[17256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679301, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.238.254@KRBTEST.COM for oryx/127.10.238.253@KRBTEST.COM
I20260929 10:55:01.814370 17859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:01.807408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.253:45709 (local address 127.0.0.1:47966)
0929 10:55:01.807550 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:01.807579 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:01.807662 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:01.807809 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:01.807813 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:01.808061 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:01.808103 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:01.808105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:01.808796 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:01.808797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:01.809442 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:01.809444 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:01.809813 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:01.809814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:01.809891 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:01.810346 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:01.810351 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:01.811777 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:55:01.813388 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:01.813394 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:01.813396 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:01.813581 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:01.813789 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:01.813790 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:01.813790 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:01.813842 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:01.814096 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:55:01.814102 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:55:01.814139 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 10:55:01.814198 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 10:55:01.814278 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":66,"threads_started":1}
I20260929 10:55:01.814414 17838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:01.807456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47966 (local address 127.10.238.253:45709)
0929 10:55:01.807490 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:01.807491 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:01.807638 (+ 147us) server_negotiation.cc:407] Connection header received
0929 10:55:01.807724 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:01.807725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:01.807734 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:01.807786 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:01.808182 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:01.808709 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:01.809548 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:01.809740 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:01.809811 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:01.811862 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:01.811881 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:01.811881 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:01.811889 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:01.813309 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:01.813654 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:01.813655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:01.813655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:01.813712 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:01.813901 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:01.813902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:01.813902 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:01.814036 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:01.814100 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:01.814243 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 10:55:01.814331 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 10:55:01.815029 17300 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.10.238.253:45709: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "59092f98066b4395b785a2698bf678c7"
member_type: NON_VOTER
last_known_addr {
host: "127.10.238.253"
port: 45709
}
attrs {
promote: true
}
}
dest_uuid: "82fdb61571ad46209f5fe7dca6fae4bf"
cas_config_opid_index: -1
I20260929 10:55:01.815336 17300 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } } peers { permanent_uuid: "59092f98066b4395b785a2698bf678c7" member_type: NON_VOTER last_known_addr { host: "127.10.238.253" port: 45709 } attrs { promote: true } }
I20260929 10:55:01.815776 17861 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 59092f98066b4395b785a2698bf678c7 (127.10.238.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } } peers { permanent_uuid: "59092f98066b4395b785a2698bf678c7" member_type: NON_VOTER last_known_addr { host: "127.10.238.253" port: 45709 } attrs { promote: true } } }
I20260929 10:55:01.816092 17807 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:55:01.816928 17861 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.10.238.253:45709
I20260929 10:55:01.817018 17807 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 [term 1 FOLLOWER]: Refusing update from remote peer 82fdb61571ad46209f5fe7dca6fae4bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 10:55:01.817142 17765 master.cc:537] Master@127.10.238.253:45709 shutting down...
I20260929 10:55:01.817235 17860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } } peers { permanent_uuid: "59092f98066b4395b785a2698bf678c7" member_type: NON_VOTER last_known_addr { host: "127.10.238.253" port: 45709 } attrs { promote: true } } }
I20260929 10:55:01.817348 17860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf [sys.catalog]: This master's current role is: LEADER
I20260929 10:55:01.817655 17860 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "59092f98066b4395b785a2698bf678c7" member_type: NON_VOTER last_known_addr { host: "127.10.238.253" port: 45709 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 10:55:01.819227 17287 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260929 10:55:01.819805 17287 consensus_peers.cc:487] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf -> Peer 59092f98066b4395b785a2698bf678c7 (127.10.238.253:45709): Couldn't send request to peer 59092f98066b4395b785a2698bf678c7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 10:55:01.820269 17765 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:55:01.820355 17765 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:55:01.820403 17765 tablet_replica.cc:330] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7: stopping tablet replica
I20260929 10:55:01.823137 17765 master.cc:552] Master@127.10.238.253:45709 shutdown complete.
I20260929 10:55:01.823181 17765 master_runner.cc:305] Clearing existing system tablet
I20260929 10:55:01.823323 17765 fs_manager.cc:359] Metadata directory not provided
I20260929 10:55:01.823369 17765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:55:01.824131 17765 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 10:55:01.824637 17865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:01.824772 17765 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:55:01.824831 17765 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:01.824856 17765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-1/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-1/wal
uuid: "59092f98066b4395b785a2698bf678c7"
format_stamp: "Formatted at 2026-09-29 10:55:01 on dist-test-slave-8nrt"
server_key: "05ddcb71b1da9d853e47c87c39cf63d2"
server_key_iv: "7908daccdafec69643aecd3e0179ceca"
server_key_version: "encryptionkey@0"
I20260929 10:55:01.825610 17765 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:55:01.827061 17765 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 10:55:01.827135 17765 log.cc:1196] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 10:55:01.827346 17765 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7: Deleting consensus metadata
I20260929 10:55:01.827463 17765 master_runner.cc:315] Copying system tablet from 127.10.238.254:39293
I20260929 10:55:01.828042 17765 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7: tablet copy: Beginning tablet copy session from remote peer at address 127.10.238.254:39293
I20260929 10:55:01.834456 17871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:01.828325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39293 (local address 127.0.0.1:54798)
0929 10:55:01.828459 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:01.828467 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:01.828515 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:01.828632 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:01.828633 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:01.828783 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:01.828829 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:01.828830 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:01.829755 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:01.829756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:01.830259 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:01.830261 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:01.830730 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:01.830730 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:01.830801 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:01.831314 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:01.831318 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:01.831613 (+ 295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:55:01.833251 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:01.833252 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:01.833252 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:01.833569 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:01.833901 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:01.833902 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:01.833902 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:01.833955 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:01.834214 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:55:01.834215 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:55:01.834247 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 10:55:01.834305 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 10:55:01.834376 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":76,"threads_started":1}
I20260929 10:55:01.834494 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:01.828377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54798 (local address 127.10.238.254:39293)
0929 10:55:01.828424 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:01.828426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:01.828522 (+ 96us) server_negotiation.cc:407] Connection header received
0929 10:55:01.828550 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:01.828550 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:01.828560 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:01.828630 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:01.828893 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:01.829669 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:01.830343 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:01.830648 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:01.830734 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:01.831700 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:01.831701 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:01.831701 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:01.831708 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:01.833170 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:01.833771 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:01.833772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:01.833773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:01.833831 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:01.834018 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:01.834019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:01.834019 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:01.834154 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:01.834219 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:01.834330 (+ 111us) server_negotiation.cc:299] Negotiation successful
0929 10:55:01.834400 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 10:55:01.835035 17321 tablet_copy_service.cc:140] P 82fdb61571ad46209f5fe7dca6fae4bf: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 59092f98066b4395b785a2698bf678c7 ({username='oryx', principal='oryx/127.10.238.253@KRBTEST.COM'} at 127.0.0.1:54798)
I20260929 10:55:01.835124 17321 tablet_copy_service.cc:161] P 82fdb61571ad46209f5fe7dca6fae4bf: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 59092f98066b4395b785a2698bf678c7 at {username='oryx', principal='oryx/127.10.238.253@KRBTEST.COM'} at 127.0.0.1:54798: session id = 59092f98066b4395b785a2698bf678c7-00000000000000000000000000000000
I20260929 10:55:01.835721 17321 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 10:55:01.836339 17765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 10:55:01.837504 17765 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7: tablet copy: Starting download of 0 data blocks...
I20260929 10:55:01.837651 17765 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7: tablet copy: Starting download of 1 WAL segments...
I20260929 10:55:01.838438 17765 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 10:55:01.839255 17321 tablet_copy_service.cc:342] P 82fdb61571ad46209f5fe7dca6fae4bf: Request end of tablet copy session 59092f98066b4395b785a2698bf678c7-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.10.238.253@KRBTEST.COM'} at 127.0.0.1:54798
I20260929 10:55:01.839324 17321 tablet_copy_service.cc:433] P 82fdb61571ad46209f5fe7dca6fae4bf: ending tablet copy session 59092f98066b4395b785a2698bf678c7-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 59092f98066b4395b785a2698bf678c7
I20260929 10:55:01.841790 17765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:55:01.842921 17872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:01.842991 17873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:01.843046 17875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:55:01.843443 17765 server_base.cc:1034] running on GCE node
I20260929 10:55:01.843536 17765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:55:01.843683 17765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:55:01.844810 17765 hybrid_clock.cc:648] HybridClock initialized: now 1790679301844802 us; error 24 us; skew 500 ppm
Sep 29 10:55:01 dist-test-slave-8nrt krb5kdc[17256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679301, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.238.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:55:01.846153 17765 init.cc:377] Logged in from keytab as oryx/127.10.238.253@KRBTEST.COM (short username oryx)
I20260929 10:55:01.846388 17765 webserver.cc:460] Webserver started at http://127.10.238.253:44133/ using document root <none> and password file <none>
I20260929 10:55:01.846493 17765 fs_manager.cc:359] Metadata directory not provided
I20260929 10:55:01.846524 17765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:55:01.847126 17765 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:01.847616 17882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:01.847744 17765 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:01.847802 17765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-1/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-1/wal
uuid: "59092f98066b4395b785a2698bf678c7"
format_stamp: "Formatted at 2026-09-29 10:55:01 on dist-test-slave-8nrt"
server_key: "05ddcb71b1da9d853e47c87c39cf63d2"
server_key_iv: "7908daccdafec69643aecd3e0179ceca"
server_key_version: "encryptionkey@0"
I20260929 10:55:01.847877 17765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:55:01.860430 17765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:55:01.862030 17765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:55:01.865422 17765 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.253:45709
I20260929 10:55:01.865662 17934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.253:45709 every 8 connection(s)
I20260929 10:55:01.865839 17765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-1/data/info.pb
I20260929 10:55:01.866750 17935 sys_catalog.cc:263] Verifying existing consensus state
I20260929 10:55:01.867224 17935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7: Bootstrap starting.
I20260929 10:55:01.870004 17935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 10:55:01.870236 17935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7: Bootstrap complete.
I20260929 10:55:01.870426 17935 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } } peers { permanent_uuid: "59092f98066b4395b785a2698bf678c7" member_type: NON_VOTER last_known_addr { host: "127.10.238.253" port: 45709 } attrs { promote: true } }
I20260929 10:55:01.870509 17935 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 59092f98066b4395b785a2698bf678c7, State: Initialized, Role: LEARNER
I20260929 10:55:01.870543 17935 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } } peers { permanent_uuid: "59092f98066b4395b785a2698bf678c7" member_type: NON_VOTER last_known_addr { host: "127.10.238.253" port: 45709 } attrs { promote: true } }
I20260929 10:55:01.870698 17935 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:55:01.870713 17938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } } peers { permanent_uuid: "59092f98066b4395b785a2698bf678c7" member_type: NON_VOTER last_known_addr { host: "127.10.238.253" port: 45709 } attrs { promote: true } } }
I20260929 10:55:01.870827 17938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 [sys.catalog]: This master's current role is: LEARNER
I20260929 10:55:01.872081 17949 catalog_manager.cc:1259] Loaded cluster ID: 8ee41e681375487bbdaa5339ed2563f0
I20260929 10:55:01.872128 17949 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7: loading cluster ID for follower catalog manager: success
I20260929 10:55:01.873135 17949 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7: acquiring CA information for follower catalog manager: success
I20260929 10:55:01.873452 17949 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 10:55:01.890931 17950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:01.884337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47982 (local address 127.10.238.253:45709)
0929 10:55:01.884504 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:01.884506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:01.884513 (+ 7us) server_negotiation.cc:407] Connection header received
0929 10:55:01.884534 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:01.884535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:01.884543 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:01.884618 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:01.885191 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:01.885963 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:01.886698 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:01.886915 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:01.886994 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:01.888264 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:01.888265 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:01.888266 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:01.888272 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:01.889763 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:01.890123 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:01.890124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:01.890124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:01.890182 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:01.890397 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:01.890398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:01.890399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:01.890503 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:01.890610 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:01.890713 (+ 103us) server_negotiation.cc:299] Negotiation successful
0929 10:55:01.890803 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":60,"threads_started":1}
WARNING: no policy specified for oryx/127.10.238.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.238.252@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.238.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.252.keytab.
Entry for principal oryx/127.10.238.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.252.keytab.
I20260929 10:55:01.918349 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.252:37593
--webserver_interface=127.10.238.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.252.keytab
--principal=oryx/127.10.238.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:46719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:39293,127.10.238.253:45709,127.10.238.252:37593
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:55:02.011476 17953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:55:02.011667 17953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:55:02.011688 17953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:55:02.012750 17953 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:55:02.012784 17953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:55:02.012797 17953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:55:02.012810 17953 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:55:02.012822 17953 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:55:02.014266 17953 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:46719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:39293,127.10.238.253:45709,127.10.238.252:37593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.252.keytab
--principal=oryx/127.10.238.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.252:37593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.238.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:55:02.014436 17953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:55:02.014608 17953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:55:02.019348 17961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:02.019356 17958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:55:02.019411 17953 server_base.cc:1034] running on GCE node
W20260929 10:55:02.019359 17959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:55:02.019882 17953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:55:02.020159 17953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:55:02.021323 17953 hybrid_clock.cc:648] HybridClock initialized: now 1790679302021311 us; error 28 us; skew 500 ppm
Sep 29 10:55:02 dist-test-slave-8nrt krb5kdc[17256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679302, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.238.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:55:02.023869 17953 init.cc:377] Logged in from keytab as oryx/127.10.238.252@KRBTEST.COM (short username oryx)
I20260929 10:55:02.024288 17953 webserver.cc:460] Webserver started at http://127.10.238.252:37073/ using document root <none> and password file <none>
I20260929 10:55:02.024550 17953 fs_manager.cc:359] Metadata directory not provided
I20260929 10:55:02.024619 17953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:55:02.024766 17953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:55:02.025882 17953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-2/data/instance:
uuid: "952082091a5949e68504685e56cd3c94"
format_stamp: "Formatted at 2026-09-29 10:55:02 on dist-test-slave-8nrt"
server_key: "d51f7efa1cfe207e051f6883570112bb"
server_key_iv: "a0ea8bd4ed43dc2456eac3161d822dde"
server_key_version: "encryptionkey@0"
I20260929 10:55:02.026263 17953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-2/wal/instance:
uuid: "952082091a5949e68504685e56cd3c94"
format_stamp: "Formatted at 2026-09-29 10:55:02 on dist-test-slave-8nrt"
server_key: "d51f7efa1cfe207e051f6883570112bb"
server_key_iv: "a0ea8bd4ed43dc2456eac3161d822dde"
server_key_version: "encryptionkey@0"
I20260929 10:55:02.027555 17953 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 10:55:02.028280 17968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:02.028458 17953 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:55:02.028555 17953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-2/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-2/wal
uuid: "952082091a5949e68504685e56cd3c94"
format_stamp: "Formatted at 2026-09-29 10:55:02 on dist-test-slave-8nrt"
server_key: "d51f7efa1cfe207e051f6883570112bb"
server_key_iv: "a0ea8bd4ed43dc2456eac3161d822dde"
server_key_version: "encryptionkey@0"
I20260929 10:55:02.028640 17953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:55:02.038339 17953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:55:02.040562 17953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:55:02.040740 17953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:55:02.044837 17953 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.252:37593
I20260929 10:55:02.044888 18020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.252:37593 every 8 connection(s)
I20260929 10:55:02.045188 17953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-2/data/info.pb
I20260929 10:55:02.046209 18021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:55:02.047780 18021 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 10:55:02 dist-test-slave-8nrt krb5kdc[17256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679302, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.238.252@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM
I20260929 10:55:02.053474 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 17953
I20260929 10:55:02.056931 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.048569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54800 (local address 127.10.238.254:39293)
0929 10:55:02.048635 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:02.048637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:02.048714 (+ 77us) server_negotiation.cc:407] Connection header received
0929 10:55:02.049215 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:02.049215 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:02.049225 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:02.049345 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:02.049920 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.050402 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.051322 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.051491 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.051538 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.053766 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:02.053767 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:02.053767 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:02.053773 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:02.055501 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:02.055914 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:02.055916 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:02.055916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:02.055980 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:02.056253 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:02.056254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:02.056254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:02.056388 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:02.056466 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:02.056750 (+ 284us) server_negotiation.cc:299] Negotiation successful
0929 10:55:02.056843 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 10:55:02.057081 18023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.048422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39293 (local address 127.0.0.1:54800)
0929 10:55:02.048589 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:02.048630 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:02.049129 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:02.049417 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:02.049420 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:02.049657 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:02.049847 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:02.049850 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.050480 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.050481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:02.051252 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:02.051254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.051591 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.051592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:02.051689 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.052203 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:02.052210 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:02.053677 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:55:02.055598 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:02.055605 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:02.055606 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:02.055835 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:02.056057 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:02.056058 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:02.056058 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:02.056170 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:02.056458 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:55:02.056464 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:55:02.056604 (+ 140us) client_negotiation.cc:769] Sending connection context
0929 10:55:02.056708 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 10:55:02.056858 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":99,"threads_started":1}
I20260929 10:55:02.057641 18021 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45709 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 10:55:02 dist-test-slave-8nrt krb5kdc[17256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.238.252@KRBTEST.COM
Sep 29 10:55:02 dist-test-slave-8nrt krb5kdc[17256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679302, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.238.252@KRBTEST.COM for oryx/127.10.238.253@KRBTEST.COM
I20260929 10:55:02.062654 18024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.054016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54858 (local address 127.10.238.252:37593)
0929 10:55:02.054207 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:02.054211 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:02.054226 (+ 15us) server_negotiation.cc:407] Connection header received
0929 10:55:02.054243 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:02.054245 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:02.054255 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:02.054379 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:02.054958 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.055646 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.056386 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.056701 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.056791 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.059561 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:02.059567 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:02.059568 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:02.059574 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:02.061321 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:02.061703 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:02.061707 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:02.061707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:02.061765 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:02.061973 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:02.061974 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:02.061974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:02.062136 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:02.062296 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:02.062444 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 10:55:02.062545 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":105,"threads_started":1}
I20260929 10:55:02.064658 18023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.057911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.253:45709 (local address 127.0.0.1:47984)
0929 10:55:02.057964 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:02.057976 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:02.058019 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:02.058203 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:02.058204 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:02.058316 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:02.058358 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:02.058359 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.059066 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.059068 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:02.059849 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:02.059851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.060377 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.060378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:02.060452 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.060801 (+ 349us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:02.060806 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:02.062156 (+ 1350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:55:02.063935 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:02.063936 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:02.063936 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:02.064083 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:02.064251 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:02.064252 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:02.064252 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:02.064299 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:02.064464 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:55:02.064465 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:55:02.064495 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 10:55:02.064542 (+ 47us) client_negotiation.cc:240] Negotiation successful
0929 10:55:02.064603 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260929 10:55:02.064774 17950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.057995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47984 (local address 127.10.238.253:45709)
0929 10:55:02.058069 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:02.058071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:02.058082 (+ 11us) server_negotiation.cc:407] Connection header received
0929 10:55:02.058116 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:02.058117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:02.058132 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:02.058196 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:02.058429 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.058969 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.059942 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.060297 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.060368 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.062239 (+ 1871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:02.062241 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:02.062241 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:02.062249 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:02.063868 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:02.064147 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:02.064147 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:02.064148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:02.064195 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:02.064335 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:02.064335 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:02.064336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:02.064418 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:02.064459 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:02.064568 (+ 109us) server_negotiation.cc:299] Negotiation successful
0929 10:55:02.064636 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 10:55:02.065119 18021 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.238.252" port: 37593 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 10:55:02 dist-test-slave-8nrt krb5kdc[17256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679302, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.238.252@KRBTEST.COM for oryx/127.10.238.252@KRBTEST.COM
I20260929 10:55:02.070649 18024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.065477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54874 (local address 127.10.238.252:37593)
0929 10:55:02.065537 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:02.065538 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:02.065546 (+ 8us) server_negotiation.cc:407] Connection header received
0929 10:55:02.065565 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:02.065566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:02.065574 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:02.065628 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:02.065806 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.066218 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.066757 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.066932 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.066979 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.068485 (+ 1506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:02.068486 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:02.068487 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:02.068492 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:02.069674 (+ 1182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:02.069959 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:02.069959 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:02.069960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:02.070005 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:02.070157 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:02.070158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:02.070158 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:02.070243 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:02.070296 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:02.070492 (+ 196us) server_negotiation.cc:299] Negotiation successful
0929 10:55:02.070585 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":44}
I20260929 10:55:02.070688 18023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.065414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.252:37593 (local address 127.0.0.1:54874)
0929 10:55:02.065463 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:02.065473 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:02.065524 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:02.065632 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:02.065633 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:02.065739 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:02.065764 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:02.065765 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.066274 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.066274 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:02.066694 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:02.066695 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.067002 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.067002 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:02.067034 (+ 32us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.067392 (+ 358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:02.067396 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:02.068415 (+ 1019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:55:02.069748 (+ 1333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:02.069748 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:02.069749 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:02.069898 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:02.070059 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:02.070060 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:02.070060 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:02.070107 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:02.070295 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:55:02.070296 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:55:02.070353 (+ 57us) client_negotiation.cc:769] Sending connection context
0929 10:55:02.070429 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 10:55:02.070532 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 10:55:02.072010 18021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94: Bootstrap starting.
I20260929 10:55:02.072803 18021 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94: Neither blocks nor log segments found. Creating new log.
I20260929 10:55:02.073041 18021 log.cc:824] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94: Log is configured to *not* fsync() on all Append() calls
I20260929 10:55:02.073732 18021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94: No bootstrap required, opened a new log
I20260929 10:55:02.075160 18021 raft_consensus.cc:348] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } } peers { permanent_uuid: "59092f98066b4395b785a2698bf678c7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45709 } } peers { permanent_uuid: "952082091a5949e68504685e56cd3c94" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 37593 } }
I20260929 10:55:02.075301 18021 raft_consensus.cc:374] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:55:02.075347 18021 raft_consensus.cc:729] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 952082091a5949e68504685e56cd3c94, State: Initialized, Role: FOLLOWER
I20260929 10:55:02.075502 18021 consensus_queue.cc:260] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } } peers { permanent_uuid: "59092f98066b4395b785a2698bf678c7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45709 } } peers { permanent_uuid: "952082091a5949e68504685e56cd3c94" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 37593 } }
I20260929 10:55:02.075745 18026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } } peers { permanent_uuid: "59092f98066b4395b785a2698bf678c7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45709 } } peers { permanent_uuid: "952082091a5949e68504685e56cd3c94" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 37593 } } }
I20260929 10:55:02.075814 18021 sys_catalog.cc:564] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:55:02.075829 18026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 10:55:02.077503 18037 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:55:02.077564 18037 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:55:02.079795 17953 master_runner.cc:428] Detected that this master 952082091a5949e68504685e56cd3c94 is joining an existing cluster
I20260929 10:55:02.079846 17953 master_runner.cc:432] Initiating AddMaster RPC to add 127.10.238.252:37593
I20260929 10:55:02.088441 18045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.081093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.252:37593 (local address 127.0.0.1:54878)
0929 10:55:02.081638 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:02.081647 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:02.081707 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:02.081866 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:02.081866 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:02.082042 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:02.082100 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:02.082102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.082975 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.082975 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:02.083788 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:02.083790 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.084259 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.084259 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:02.084317 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.084717 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:02.084735 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:02.085189 (+ 454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:55:02.087006 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:02.087007 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:02.087007 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:02.087459 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:02.087688 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:02.087689 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:02.087689 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:02.087763 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:02.088082 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:55:02.088083 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:55:02.088138 (+ 55us) client_negotiation.cc:769] Sending connection context
0929 10:55:02.088215 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 10:55:02.088315 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":57,"threads_started":1}
I20260929 10:55:02.088510 18024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.081219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54878 (local address 127.10.238.252:37593)
0929 10:55:02.081338 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:02.081339 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:02.081689 (+ 350us) server_negotiation.cc:407] Connection header received
0929 10:55:02.081781 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:02.081782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:02.081791 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:02.081816 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:02.082176 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.082884 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.083878 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.084147 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.084182 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.085352 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:02.085353 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:02.085353 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:02.085361 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:02.086922 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:02.087554 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:02.087555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:02.087555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:02.087612 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:02.087832 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:02.087833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:02.087833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:02.087990 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:02.088073 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:02.088260 (+ 187us) server_negotiation.cc:299] Negotiation successful
0929 10:55:02.088378 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":103}
I20260929 10:55:02.092921 18026 raft_consensus.cc:482] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:55:02.093006 18026 raft_consensus.cc:504] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } } peers { permanent_uuid: "59092f98066b4395b785a2698bf678c7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45709 } } peers { permanent_uuid: "952082091a5949e68504685e56cd3c94" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 37593 } }
I20260929 10:55:02.093225 18026 leader_election.cc:288] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82fdb61571ad46209f5fe7dca6fae4bf (127.10.238.254:39293), 59092f98066b4395b785a2698bf678c7 (127.10.238.253:45709)
I20260929 10:55:02.093415 17311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "952082091a5949e68504685e56cd3c94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" is_pre_election: true
I20260929 10:55:02.093438 17909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "952082091a5949e68504685e56cd3c94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59092f98066b4395b785a2698bf678c7" is_pre_election: true
I20260929 10:55:02.093528 17909 raft_consensus.cc:1799] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 [term 1 LEARNER]: Handling vote request from an unknown peer 952082091a5949e68504685e56cd3c94
I20260929 10:55:02.093566 17909 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate 952082091a5949e68504685e56cd3c94 for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260929 10:55:02.093753 17970 leader_election.cc:302] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 952082091a5949e68504685e56cd3c94; no voters: 59092f98066b4395b785a2698bf678c7, 82fdb61571ad46209f5fe7dca6fae4bf
I20260929 10:55:02.093897 18026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:55:02.094766 18026 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 10:55:02.105356 17950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.081073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47998 (local address 127.10.238.253:45709)
0929 10:55:02.081164 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:02.081166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:02.081174 (+ 8us) server_negotiation.cc:407] Connection header received
0929 10:55:02.081197 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:02.081198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:02.081210 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:02.081330 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:02.081933 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.082692 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.083935 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.084225 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.084264 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.102801 (+ 18537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:02.102802 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:02.102802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:02.102809 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:02.104218 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:02.104592 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:02.104594 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:02.104594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:02.104653 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:02.104819 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:02.104820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:02.104820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:02.104944 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:02.105011 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:02.105157 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 10:55:02.105233 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 10:55:02.105365 18043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.080869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.253:45709 (local address 127.0.0.1:47998)
0929 10:55:02.081054 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:02.081068 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:02.081133 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:02.081530 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:02.081531 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:02.081768 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:02.081844 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:02.081846 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.082781 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.082782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:02.083858 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:02.083859 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.084346 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.084347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:02.084402 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.102324 (+ 17922us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:02.102328 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:02.102715 (+ 387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:55:02.104305 (+ 1590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:02.104307 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:02.104307 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:02.104522 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:02.104705 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:02.104705 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:02.104706 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:02.104766 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:02.105013 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:55:02.105015 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:55:02.105055 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 10:55:02.105133 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 10:55:02.105233 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":48,"spinlock_wait_cycles":9472,"thread_start_us":118,"threads_started":1}
I20260929 10:55:02.105382 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.081122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54802 (local address 127.10.238.254:39293)
0929 10:55:02.081383 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:02.081385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:02.081393 (+ 8us) server_negotiation.cc:407] Connection header received
0929 10:55:02.081415 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:02.081416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:02.081425 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:02.081485 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:02.081911 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.082680 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.083833 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.084124 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.084161 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.102644 (+ 18483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:02.102646 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:02.102646 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:02.102654 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:02.104245 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:02.104639 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:02.104640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:02.104640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:02.104692 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:02.104858 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:02.104859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:02.104859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:02.104984 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:02.105040 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:02.105201 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 10:55:02.105305 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230}
I20260929 10:55:02.105625 18044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.080870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39293 (local address 127.0.0.1:54802)
0929 10:55:02.081280 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:02.081289 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:02.081330 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:02.081499 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:02.081500 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:02.081714 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:02.081803 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:02.081805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.082781 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.082782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:02.083729 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:02.083731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.084250 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.084251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:02.084316 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.101969 (+ 17653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:02.101976 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:02.102556 (+ 580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:55:02.104336 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:02.104337 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:02.104338 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:02.104580 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:02.104742 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:02.104742 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:02.104743 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:02.104807 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:02.105065 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:55:02.105066 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:55:02.105103 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 10:55:02.105446 (+ 343us) client_negotiation.cc:240] Negotiation successful
0929 10:55:02.105535 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":42,"thread_start_us":96,"threads_started":1}
Sep 29 10:55:02 dist-test-slave-8nrt krb5kdc[17256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679301, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.238.254@KRBTEST.COM for oryx/127.10.238.252@KRBTEST.COM
I20260929 10:55:02.113590 17859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.107437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.252:37593 (local address 127.0.0.1:54880)
0929 10:55:02.107474 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:02.107484 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:02.107520 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:02.107633 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:02.107633 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:02.107774 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:02.107813 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:02.107815 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.108404 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.108405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:02.108873 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:02.108874 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.109223 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.109224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:02.109337 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.109755 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:02.109759 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:02.110989 (+ 1230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:55:02.112524 (+ 1535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:02.112525 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:02.112525 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:02.112692 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:02.112900 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:02.112901 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:02.112902 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:02.112958 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:02.113217 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:55:02.113218 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:55:02.113276 (+ 58us) client_negotiation.cc:769] Sending connection context
0929 10:55:02.113387 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 10:55:02.113498 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260929 10:55:02.113883 18024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.107487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54880 (local address 127.10.238.252:37593)
0929 10:55:02.107526 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:02.107528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:02.107551 (+ 23us) server_negotiation.cc:407] Connection header received
0929 10:55:02.107580 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:02.107581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:02.107589 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:02.107632 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:02.107886 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.108329 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.108957 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.109155 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.109225 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.111078 (+ 1853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:02.111080 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:02.111080 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:02.111087 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:02.112423 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:02.112773 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:02.112774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:02.112775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:02.112836 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:02.113036 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:02.113038 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:02.113038 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:02.113142 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:02.113211 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:02.113412 (+ 201us) server_negotiation.cc:299] Negotiation successful
0929 10:55:02.113496 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 10:55:02.113895 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 17363
I20260929 10:55:02.117939 17300 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.10.238.252:37593: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "952082091a5949e68504685e56cd3c94"
member_type: NON_VOTER
last_known_addr {
host: "127.10.238.252"
port: 37593
}
attrs {
promote: true
}
}
dest_uuid: "82fdb61571ad46209f5fe7dca6fae4bf"
cas_config_opid_index: 5
I20260929 10:55:02.118181 17300 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } } peers { permanent_uuid: "59092f98066b4395b785a2698bf678c7" member_type: NON_VOTER last_known_addr { host: "127.10.238.253" port: 45709 } attrs { promote: true } } peers { permanent_uuid: "952082091a5949e68504685e56cd3c94" member_type: NON_VOTER last_known_addr { host: "127.10.238.252" port: 37593 } attrs { promote: true } }
I20260929 10:55:02.118485 17860 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 952082091a5949e68504685e56cd3c94 (127.10.238.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } } peers { permanent_uuid: "59092f98066b4395b785a2698bf678c7" member_type: NON_VOTER last_known_addr { host: "127.10.238.253" port: 45709 } attrs { promote: true } } peers { permanent_uuid: "952082091a5949e68504685e56cd3c94" member_type: NON_VOTER last_known_addr { host: "127.10.238.252" port: 37593 } attrs { promote: true } } }
I20260929 10:55:02.119489 17860 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.10.238.252:37593
I20260929 10:55:02.119562 17860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } } peers { permanent_uuid: "59092f98066b4395b785a2698bf678c7" member_type: NON_VOTER last_known_addr { host: "127.10.238.253" port: 45709 } attrs { promote: true } } peers { permanent_uuid: "952082091a5949e68504685e56cd3c94" member_type: NON_VOTER last_known_addr { host: "127.10.238.252" port: 37593 } attrs { promote: true } } }
I20260929 10:55:02.119634 17860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf [sys.catalog]: This master's current role is: LEADER
I20260929 10:55:02.119853 17953 master.cc:537] Master@127.10.238.252:37593 shutting down...
W20260929 10:55:02.120441 17287 consensus_peers.cc:487] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf -> Peer 952082091a5949e68504685e56cd3c94 (127.10.238.252:37593): Couldn't send request to peer 952082091a5949e68504685e56cd3c94. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 10:55:02.120708 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.193:42961
--local_ip_for_outbound_sockets=127.10.238.193
--tserver_master_addrs=127.10.238.254:39293
--webserver_port=43063
--webserver_interface=127.10.238.193
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.193.keytab
--principal=oryx/127.10.238.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:46719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.10.238.254:39293,127.10.238.253:45709
--tserver_master_addrs=127.10.238.254:39293,127.10.238.253:45709,127.10.238.252:37593 with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:55:02.123158 17953 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:55:02.123250 17953 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:55:02.123296 17953 tablet_replica.cc:330] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94: stopping tablet replica
I20260929 10:55:02.124770 17859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.119884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.253:45709 (local address 127.0.0.1:48000)
0929 10:55:02.120000 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:02.120011 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:02.120058 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:02.120323 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:02.120324 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:02.120326 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:55:02.120364 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:02.120365 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.121960 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.121961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:02.123379 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:02.123382 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.124526 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.124527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:02.124589 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.124608 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 10:55:02.124655 (+ 47us) client_negotiation.cc:240] Negotiation successful
0929 10:55:02.124691 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260929 10:55:02.125240 17950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.120021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48000 (local address 127.10.238.253:45709)
0929 10:55:02.120136 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:02.120138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:02.120146 (+ 8us) server_negotiation.cc:407] Connection header received
0929 10:55:02.120169 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:02.120171 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:02.120194 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:02.120250 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:55:02.121083 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.121861 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.123554 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.124351 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.124419 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.124493 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:02.125059 (+ 566us) server_negotiation.cc:299] Negotiation successful
0929 10:55:02.125178 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260929 10:55:02.125715 17909 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 [term 1 LEARNER]: Refusing update from remote peer 82fdb61571ad46209f5fe7dca6fae4bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 10:55:02.125937 17861 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "59092f98066b4395b785a2698bf678c7" member_type: NON_VOTER last_known_addr { host: "127.10.238.253" port: 45709 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 10:55:02.126531 17909 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 952082091a5949e68504685e56cd3c94 (127.10.238.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } } peers { permanent_uuid: "59092f98066b4395b785a2698bf678c7" member_type: NON_VOTER last_known_addr { host: "127.10.238.253" port: 45709 } attrs { promote: true } } peers { permanent_uuid: "952082091a5949e68504685e56cd3c94" member_type: NON_VOTER last_known_addr { host: "127.10.238.252" port: 37593 } attrs { promote: true } } }
I20260929 10:55:02.128008 17861 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf: attempting to promote NON_VOTER 59092f98066b4395b785a2698bf678c7 to VOTER
I20260929 10:55:02.128098 17938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82fdb61571ad46209f5fe7dca6fae4bf. Latest consensus state: current_term: 1 leader_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } } peers { permanent_uuid: "59092f98066b4395b785a2698bf678c7" member_type: NON_VOTER last_known_addr { host: "127.10.238.253" port: 45709 } attrs { promote: true } } peers { permanent_uuid: "952082091a5949e68504685e56cd3c94" member_type: NON_VOTER last_known_addr { host: "127.10.238.252" port: 37593 } attrs { promote: true } } }
I20260929 10:55:02.128226 17938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 [sys.catalog]: This master's current role is: LEARNER
I20260929 10:55:02.128183 17861 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } } peers { permanent_uuid: "59092f98066b4395b785a2698bf678c7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45709 } attrs { promote: false } } peers { permanent_uuid: "952082091a5949e68504685e56cd3c94" member_type: NON_VOTER last_known_addr { host: "127.10.238.252" port: 37593 } attrs { promote: true } }
I20260929 10:55:02.128273 17938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } } peers { permanent_uuid: "59092f98066b4395b785a2698bf678c7" member_type: NON_VOTER last_known_addr { host: "127.10.238.253" port: 45709 } attrs { promote: true } } peers { permanent_uuid: "952082091a5949e68504685e56cd3c94" member_type: NON_VOTER last_known_addr { host: "127.10.238.252" port: 37593 } attrs { promote: true } } }
I20260929 10:55:02.128331 17938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 [sys.catalog]: This master's current role is: LEARNER
I20260929 10:55:02.128921 17859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.128771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.252:37593 (local address 127.0.0.1:54888)
0929 10:55:02.128837 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:02.128886 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.252:37593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 10:55:02.128921 17909 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 [term 1 LEARNER]: Refusing update from remote peer 82fdb61571ad46209f5fe7dca6fae4bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260929 10:55:02.129338 17859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.129230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.252:37593 (local address 127.0.0.1:54896)
0929 10:55:02.129282 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:02.129311 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.252:37593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260929 10:55:02.129191 18049 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "59092f98066b4395b785a2698bf678c7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45709 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260929 10:55:02.129549 17287 consensus_peers.cc:487] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf -> Peer 952082091a5949e68504685e56cd3c94 (127.10.238.252:37593): Couldn't send request to peer 952082091a5949e68504685e56cd3c94. Status: Network error: Client connection negotiation failed: client connection to 127.10.238.252:37593: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 10:55:02.130216 17860 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 59092f98066b4395b785a2698bf678c7 (127.10.238.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } } peers { permanent_uuid: "59092f98066b4395b785a2698bf678c7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45709 } attrs { promote: false } } peers { permanent_uuid: "952082091a5949e68504685e56cd3c94" member_type: NON_VOTER last_known_addr { host: "127.10.238.252" port: 37593 } attrs { promote: true } } }
I20260929 10:55:02.130395 17909 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 59092f98066b4395b785a2698bf678c7 (127.10.238.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } } peers { permanent_uuid: "59092f98066b4395b785a2698bf678c7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45709 } attrs { promote: false } } peers { permanent_uuid: "952082091a5949e68504685e56cd3c94" member_type: NON_VOTER last_known_addr { host: "127.10.238.252" port: 37593 } attrs { promote: true } } }
I20260929 10:55:02.131135 17860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } } peers { permanent_uuid: "59092f98066b4395b785a2698bf678c7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45709 } attrs { promote: false } } peers { permanent_uuid: "952082091a5949e68504685e56cd3c94" member_type: NON_VOTER last_known_addr { host: "127.10.238.252" port: 37593 } attrs { promote: true } } }
I20260929 10:55:02.131206 17860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf [sys.catalog]: This master's current role is: LEADER
I20260929 10:55:02.131251 17860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } } peers { permanent_uuid: "59092f98066b4395b785a2698bf678c7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45709 } attrs { promote: false } } peers { permanent_uuid: "952082091a5949e68504685e56cd3c94" member_type: NON_VOTER last_known_addr { host: "127.10.238.252" port: 37593 } attrs { promote: true } } }
I20260929 10:55:02.131291 17860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf [sys.catalog]: This master's current role is: LEADER
I20260929 10:55:02.131520 18048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } } peers { permanent_uuid: "59092f98066b4395b785a2698bf678c7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45709 } attrs { promote: false } } peers { permanent_uuid: "952082091a5949e68504685e56cd3c94" member_type: NON_VOTER last_known_addr { host: "127.10.238.252" port: 37593 } attrs { promote: true } } }
I20260929 10:55:02.131660 18048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:55:02.136626 17953 master.cc:552] Master@127.10.238.252:37593 shutdown complete.
I20260929 10:55:02.136682 17953 master_runner.cc:305] Clearing existing system tablet
I20260929 10:55:02.136822 17953 fs_manager.cc:359] Metadata directory not provided
I20260929 10:55:02.136869 17953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:55:02.137713 17953 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 10:55:02.138254 18052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:02.138412 17953 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:55:02.138477 17953 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 10:55:02.138505 17953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-2/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-2/wal
uuid: "952082091a5949e68504685e56cd3c94"
format_stamp: "Formatted at 2026-09-29 10:55:02 on dist-test-slave-8nrt"
server_key: "d51f7efa1cfe207e051f6883570112bb"
server_key_iv: "a0ea8bd4ed43dc2456eac3161d822dde"
server_key_version: "encryptionkey@0"
I20260929 10:55:02.139272 17953 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:55:02.140672 17953 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 10:55:02.140743 17953 log.cc:1196] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 10:55:02.140942 17953 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94: Deleting consensus metadata
I20260929 10:55:02.141073 17953 master_runner.cc:315] Copying system tablet from 127.10.238.254:39293
I20260929 10:55:02.141693 17953 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94: tablet copy: Beginning tablet copy session from remote peer at address 127.10.238.254:39293
I20260929 10:55:02.149106 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.142053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54816 (local address 127.10.238.254:39293)
0929 10:55:02.142108 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:02.142111 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:02.142222 (+ 111us) server_negotiation.cc:407] Connection header received
0929 10:55:02.142283 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:02.142283 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:02.142295 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:02.142352 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:02.142680 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.143458 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.144552 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.144851 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.144924 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.146197 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:02.146198 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:02.146199 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:02.146206 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:02.147874 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:02.148293 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:02.148295 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:02.148295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:02.148361 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:02.148575 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:02.148576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:02.148576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:02.148705 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:02.148770 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:02.148938 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 10:55:02.149014 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260929 10:55:02.149134 18058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.141992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39293 (local address 127.0.0.1:54816)
0929 10:55:02.142148 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:02.142158 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:02.142216 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:02.142362 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:02.142362 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:02.142563 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:02.142611 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:02.142612 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.143564 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.143565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:02.144457 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:02.144460 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.144934 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.144935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:02.145014 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.145703 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:02.145708 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:02.146101 (+ 393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:55:02.147991 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:02.147992 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:02.147993 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:02.148215 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:02.148437 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:02.148438 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:02.148438 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:02.148509 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:02.148776 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:55:02.148777 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:55:02.148826 (+ 49us) client_negotiation.cc:769] Sending connection context
0929 10:55:02.148900 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 10:55:02.149002 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":60,"threads_started":1}
I20260929 10:55:02.149803 17321 tablet_copy_service.cc:140] P 82fdb61571ad46209f5fe7dca6fae4bf: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 952082091a5949e68504685e56cd3c94 ({username='oryx', principal='oryx/127.10.238.252@KRBTEST.COM'} at 127.0.0.1:54816)
I20260929 10:55:02.149868 17321 tablet_copy_service.cc:161] P 82fdb61571ad46209f5fe7dca6fae4bf: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 952082091a5949e68504685e56cd3c94 at {username='oryx', principal='oryx/127.10.238.252@KRBTEST.COM'} at 127.0.0.1:54816: session id = 952082091a5949e68504685e56cd3c94-00000000000000000000000000000000
I20260929 10:55:02.150465 17321 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 10:55:02.150977 17953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 10:55:02.152370 17953 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94: tablet copy: Starting download of 0 data blocks...
I20260929 10:55:02.152595 17953 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94: tablet copy: Starting download of 1 WAL segments...
I20260929 10:55:02.153538 17953 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 10:55:02.154493 17321 tablet_copy_service.cc:342] P 82fdb61571ad46209f5fe7dca6fae4bf: Request end of tablet copy session 952082091a5949e68504685e56cd3c94-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.10.238.252@KRBTEST.COM'} at 127.0.0.1:54816
I20260929 10:55:02.154547 17321 tablet_copy_service.cc:433] P 82fdb61571ad46209f5fe7dca6fae4bf: ending tablet copy session 952082091a5949e68504685e56cd3c94-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 952082091a5949e68504685e56cd3c94
I20260929 10:55:02.157513 17953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:55:02.158741 18060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:02.158783 18059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:02.158969 18062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:55:02.163869 17953 server_base.cc:1034] running on GCE node
I20260929 10:55:02.163985 17953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:55:02.164135 17953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:55:02.165293 17953 hybrid_clock.cc:648] HybridClock initialized: now 1790679302165266 us; error 41 us; skew 500 ppm
Sep 29 10:55:02 dist-test-slave-8nrt krb5kdc[17256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679302, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.238.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:55:02.166827 17953 init.cc:377] Logged in from keytab as oryx/127.10.238.252@KRBTEST.COM (short username oryx)
I20260929 10:55:02.167090 17953 webserver.cc:460] Webserver started at http://127.10.238.252:35601/ using document root <none> and password file <none>
I20260929 10:55:02.167215 17953 fs_manager.cc:359] Metadata directory not provided
I20260929 10:55:02.167258 17953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:55:02.168154 17953 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:55:02.168841 18069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:02.169023 17953 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:55:02.169095 17953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-2/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-2/wal
uuid: "952082091a5949e68504685e56cd3c94"
format_stamp: "Formatted at 2026-09-29 10:55:02 on dist-test-slave-8nrt"
server_key: "d51f7efa1cfe207e051f6883570112bb"
server_key_iv: "a0ea8bd4ed43dc2456eac3161d822dde"
server_key_version: "encryptionkey@0"
I20260929 10:55:02.169154 17953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:55:02.185853 17953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:55:02.187660 17953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:55:02.190761 17953 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.252:37593
I20260929 10:55:02.191116 17953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/master-2/data/info.pb
I20260929 10:55:02.192072 18122 sys_catalog.cc:263] Verifying existing consensus state
I20260929 10:55:02.192544 18122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94: Bootstrap starting.
I20260929 10:55:02.193370 18121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.252:37593 every 8 connection(s)
I20260929 10:55:02.195470 18122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 10:55:02.195686 18122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94: Bootstrap complete.
I20260929 10:55:02.195950 18122 raft_consensus.cc:348] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } } peers { permanent_uuid: "59092f98066b4395b785a2698bf678c7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45709 } attrs { promote: false } } peers { permanent_uuid: "952082091a5949e68504685e56cd3c94" member_type: NON_VOTER last_known_addr { host: "127.10.238.252" port: 37593 } attrs { promote: true } }
I20260929 10:55:02.196046 18122 raft_consensus.cc:729] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 952082091a5949e68504685e56cd3c94, State: Initialized, Role: LEARNER
I20260929 10:55:02.196091 18122 consensus_queue.cc:260] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } } peers { permanent_uuid: "59092f98066b4395b785a2698bf678c7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45709 } attrs { promote: false } } peers { permanent_uuid: "952082091a5949e68504685e56cd3c94" member_type: NON_VOTER last_known_addr { host: "127.10.238.252" port: 37593 } attrs { promote: true } }
I20260929 10:55:02.196257 18122 sys_catalog.cc:564] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:55:02.196267 18125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } } peers { permanent_uuid: "59092f98066b4395b785a2698bf678c7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45709 } attrs { promote: false } } peers { permanent_uuid: "952082091a5949e68504685e56cd3c94" member_type: NON_VOTER last_known_addr { host: "127.10.238.252" port: 37593 } attrs { promote: true } } }
I20260929 10:55:02.196605 18125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94 [sys.catalog]: This master's current role is: LEARNER
I20260929 10:55:02.197760 18136 catalog_manager.cc:1259] Loaded cluster ID: 8ee41e681375487bbdaa5339ed2563f0
I20260929 10:55:02.197960 18136 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94: loading cluster ID for follower catalog manager: success
I20260929 10:55:02.199113 18136 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94: acquiring CA information for follower catalog manager: success
I20260929 10:55:02.199474 18136 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 10:55:02.223146 18046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:55:02.223327 18046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:55:02.223348 18046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:55:02.224622 18046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:55:02.224660 18046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:55:02.224684 18046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.193
I20260929 10:55:02.226235 18046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:46719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.193.keytab
--principal=oryx/127.10.238.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.193:42961
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.238.193
--webserver_port=43063
--tserver_master_addrs=127.10.238.254:39293,127.10.238.253:45709,127.10.238.252:37593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.193
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:55:02.226472 18046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:55:02.226838 18046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:55:02.231980 18141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:02.231986 18142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:02.232044 18144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:55:02.232337 18046 server_base.cc:1034] running on GCE node
I20260929 10:55:02.232564 18046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:55:02.232805 18046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:55:02.233927 18046 hybrid_clock.cc:648] HybridClock initialized: now 1790679302233901 us; error 39 us; skew 500 ppm
Sep 29 10:55:02 dist-test-slave-8nrt krb5kdc[17256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679302, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.238.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:55:02.236390 18046 init.cc:377] Logged in from keytab as oryx/127.10.238.193@KRBTEST.COM (short username oryx)
I20260929 10:55:02.236796 18046 webserver.cc:460] Webserver started at http://127.10.238.193:43063/ using document root <none> and password file <none>
I20260929 10:55:02.237051 18046 fs_manager.cc:359] Metadata directory not provided
I20260929 10:55:02.237107 18046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:55:02.238459 18046 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:55:02.239032 18151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:02.239214 18046 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:55:02.239307 18046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-0/wal
uuid: "2b18011dc9b24cf587c3bb2237afc7e6"
format_stamp: "Formatted at 2026-09-29 10:55:01 on dist-test-slave-8nrt"
server_key: "b2057d7f0337f4cab3433171432cb795"
server_key_iv: "bdc6f0ba15c1ecde67411c421764c0fe"
server_key_version: "encryptionkey@0"
I20260929 10:55:02.239639 18046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:55:02.255153 18046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:55:02.257086 18046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:55:02.257248 18046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:55:02.257504 18046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:55:02.257740 18046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:55:02.257769 18046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:02.257790 18046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:55:02.257805 18046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:02.263419 18046 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.193:42961
I20260929 10:55:02.263496 18264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.193:42961 every 8 connection(s)
I20260929 10:55:02.263841 18046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
I20260929 10:55:02.267475 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 18046
I20260929 10:55:02.267655 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 17497
Sep 29 10:55:02 dist-test-slave-8nrt krb5kdc[17256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679302, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.238.193@KRBTEST.COM for oryx/127.10.238.253@KRBTEST.COM
Sep 29 10:55:02 dist-test-slave-8nrt krb5kdc[17256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679302, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.238.193@KRBTEST.COM for oryx/127.10.238.252@KRBTEST.COM
I20260929 10:55:02.275079 17950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.264920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:47233 (local address 127.10.238.253:45709)
0929 10:55:02.264963 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:02.264965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:02.265408 (+ 443us) server_negotiation.cc:407] Connection header received
0929 10:55:02.265600 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:02.265601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:02.265613 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:02.265645 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:02.266537 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.267040 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.268494 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.268848 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.268924 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.271665 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:02.271666 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:02.271666 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:02.271674 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:02.273429 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:02.273916 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:02.273918 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:02.273918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:02.273985 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:02.274277 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:02.274278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:02.274279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:02.274461 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:02.274529 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:02.274848 (+ 319us) server_negotiation.cc:299] Negotiation successful
0929 10:55:02.274936 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 10:55:02.275238 18271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.264852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.253:45709 (local address 127.10.238.193:47233)
0929 10:55:02.265336 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:02.265346 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:02.265491 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:02.266068 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:02.266068 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:02.266238 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:02.266435 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:02.266439 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.267161 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.267162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:02.268406 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:02.268408 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.268948 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.268949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:02.269080 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.269617 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:02.269625 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:02.271556 (+ 1931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:55:02.273514 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:02.273521 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:02.273524 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:02.273805 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:02.274061 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:02.274062 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:02.274062 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:02.274198 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:02.274536 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:55:02.274542 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:55:02.274700 (+ 158us) client_negotiation.cc:769] Sending connection context
0929 10:55:02.274817 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 10:55:02.274988 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":76,"threads_started":1}
I20260929 10:55:02.275709 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.194:40983
--local_ip_for_outbound_sockets=127.10.238.194
--tserver_master_addrs=127.10.238.254:39293
--webserver_port=41693
--webserver_interface=127.10.238.194
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.194.keytab
--principal=oryx/127.10.238.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:46719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.10.238.254:39293,127.10.238.253:45709
--tserver_master_addrs=127.10.238.254:39293,127.10.238.253:45709,127.10.238.252:37593 with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:55:02.275767 18267 heartbeater.cc:347] Connected to a master server at 127.10.238.253:45709
I20260929 10:55:02.275857 18267 heartbeater.cc:464] Registering TS with master...
I20260929 10:55:02.276024 18269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.264536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:54923 (local address 127.10.238.252:37593)
0929 10:55:02.264674 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:02.264676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:02.264899 (+ 223us) server_negotiation.cc:407] Connection header received
0929 10:55:02.265704 (+ 805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:02.265705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:02.265719 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:02.265796 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:02.268181 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.269071 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.270216 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.270538 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.270630 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.272773 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:02.272775 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:02.272775 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:02.272782 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:02.274475 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:02.274852 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:02.274853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:02.274853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:02.274910 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:02.275100 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:02.275101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:02.275101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:02.275549 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:02.275593 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:02.275832 (+ 239us) server_negotiation.cc:299] Negotiation successful
0929 10:55:02.275934 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":57,"threads_started":1}
I20260929 10:55:02.276055 18267 heartbeater.cc:511] Master 127.10.238.253:45709 requested a full tablet report, sending...
I20260929 10:55:02.276293 18270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.264556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.252:37593 (local address 127.10.238.193:54923)
0929 10:55:02.264796 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:02.264824 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:02.265473 (+ 649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:02.265819 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:02.265819 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:02.266224 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:02.266436 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:02.266441 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.269154 (+ 2713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.269156 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:02.270133 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:02.270136 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.270633 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.270634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:02.270702 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.271199 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:02.271204 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:02.272700 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:55:02.274569 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:02.274569 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:02.274570 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:02.274758 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:02.274982 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:02.274983 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:02.274983 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:02.275035 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:02.275641 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:55:02.275642 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:55:02.275678 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 10:55:02.276105 (+ 427us) client_negotiation.cc:240] Negotiation successful
0929 10:55:02.276195 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":3,"thread_start_us":86,"threads_started":1}
I20260929 10:55:02.276595 18265 heartbeater.cc:347] Connected to a master server at 127.10.238.252:37593
I20260929 10:55:02.276633 18265 heartbeater.cc:464] Registering TS with master...
I20260929 10:55:02.276643 17899 ts_manager.cc:194] Registered new tserver with Master: 2b18011dc9b24cf587c3bb2237afc7e6 (127.10.238.193:42961)
I20260929 10:55:02.276690 18265 heartbeater.cc:511] Master 127.10.238.252:37593 requested a full tablet report, sending...
I20260929 10:55:02.277060 18086 ts_manager.cc:194] Registered new tserver with Master: 2b18011dc9b24cf587c3bb2237afc7e6 (127.10.238.193:42961)
Sep 29 10:55:02 dist-test-slave-8nrt krb5kdc[17256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679302, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.238.193@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM
I20260929 10:55:02.282831 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.264658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:42553 (local address 127.10.238.254:39293)
0929 10:55:02.264705 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:02.264706 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:02.265033 (+ 327us) server_negotiation.cc:407] Connection header received
0929 10:55:02.265571 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:02.265573 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:02.265587 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:02.265651 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:02.267172 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.267922 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.278171 (+ 10249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.278370 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.278432 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.280234 (+ 1802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:02.280236 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:02.280236 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:02.280244 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:02.281682 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:02.282041 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:02.282042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:02.282043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:02.282102 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:02.282301 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:02.282302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:02.282302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:02.282432 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:02.282511 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:02.282663 (+ 152us) server_negotiation.cc:299] Negotiation successful
0929 10:55:02.282737 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 10:55:02.283046 18272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.264557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39293 (local address 127.10.238.193:42553)
0929 10:55:02.264972 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:02.264981 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:02.265474 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:02.265714 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:02.265717 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:02.266238 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:02.266436 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:02.266441 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.277186 (+ 10745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.277187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:02.278086 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:02.278089 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.278445 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.278446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:02.278493 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.278899 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:02.278904 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:02.280132 (+ 1228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:55:02.281781 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:02.281781 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:02.281782 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:02.281969 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:02.282175 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:02.282176 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:02.282176 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:02.282235 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:02.282503 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:55:02.282504 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:55:02.282544 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 10:55:02.282636 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 10:55:02.282735 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":9,"thread_start_us":98,"threads_started":1}
I20260929 10:55:02.283494 18266 heartbeater.cc:347] Connected to a master server at 127.10.238.254:39293
I20260929 10:55:02.283540 18266 heartbeater.cc:464] Registering TS with master...
I20260929 10:55:02.283613 18266 heartbeater.cc:511] Master 127.10.238.254:39293 requested a full tablet report, sending...
I20260929 10:55:02.283906 17300 ts_manager.cc:194] Re-registered known tserver with Master: 2b18011dc9b24cf587c3bb2237afc7e6 (127.10.238.193:42961)
I20260929 10:55:02.284447 17300 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.238.193@KRBTEST.COM'} at 127.10.238.193:42553
W20260929 10:55:02.359993 18273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:55:02.360188 18273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:55:02.360210 18273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:55:02.361409 18273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:55:02.361449 18273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:55:02.361475 18273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.194
I20260929 10:55:02.362987 18273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:46719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.194.keytab
--principal=oryx/127.10.238.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.194:40983
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.238.194
--webserver_port=41693
--tserver_master_addrs=127.10.238.254:39293,127.10.238.253:45709,127.10.238.252:37593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.194
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:55:02.363175 18273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:55:02.363353 18273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:55:02.368218 18279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:02.368247 18281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:55:02.368256 18273 server_base.cc:1034] running on GCE node
W20260929 10:55:02.368218 18278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:55:02.368650 18273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:55:02.368916 18273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:55:02.370064 18273 hybrid_clock.cc:648] HybridClock initialized: now 1790679302370030 us; error 51 us; skew 500 ppm
Sep 29 10:55:02 dist-test-slave-8nrt krb5kdc[17256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679302, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.238.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:55:02.372846 18273 init.cc:377] Logged in from keytab as oryx/127.10.238.194@KRBTEST.COM (short username oryx)
I20260929 10:55:02.373422 18273 webserver.cc:460] Webserver started at http://127.10.238.194:41693/ using document root <none> and password file <none>
I20260929 10:55:02.373656 18273 fs_manager.cc:359] Metadata directory not provided
I20260929 10:55:02.373710 18273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:55:02.374974 18273 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 10:55:02.375473 18288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:02.375686 18273 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 10:55:02.375785 18273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-1/wal
uuid: "bb2eb90f68da49f7b9fb25af83502e1f"
format_stamp: "Formatted at 2026-09-29 10:55:01 on dist-test-slave-8nrt"
server_key: "e6e721703a907ea8d01f2a9ef03bac35"
server_key_iv: "9977ed11a932968af2e929a98e8c5e6e"
server_key_version: "encryptionkey@0"
I20260929 10:55:02.376088 18273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:55:02.385681 18273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:55:02.387394 18273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:55:02.387497 18273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:55:02.387660 18273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:55:02.387957 18273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:55:02.387988 18273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:02.388010 18273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:55:02.388055 18273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:02.393887 18273 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.194:40983
I20260929 10:55:02.393986 18401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.194:40983 every 8 connection(s)
I20260929 10:55:02.394300 18273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
I20260929 10:55:02.401216 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 18273
I20260929 10:55:02.401410 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 17631
Sep 29 10:55:02 dist-test-slave-8nrt krb5kdc[17256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679302, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.238.194@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM
Sep 29 10:55:02 dist-test-slave-8nrt krb5kdc[17256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679302, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.238.194@KRBTEST.COM for oryx/127.10.238.253@KRBTEST.COM
Sep 29 10:55:02 dist-test-slave-8nrt krb5kdc[17256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679302, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.238.194@KRBTEST.COM for oryx/127.10.238.252@KRBTEST.COM
I20260929 10:55:02.406821 17950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.395044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:58025 (local address 127.10.238.253:45709)
0929 10:55:02.395126 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:02.395128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:02.395503 (+ 375us) server_negotiation.cc:407] Connection header received
0929 10:55:02.396664 (+ 1161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:02.396666 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:02.396680 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:02.396714 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:02.397702 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.398504 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.399611 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.399908 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.399948 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.402783 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:02.402784 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:02.402784 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:02.402794 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:02.404658 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:02.405658 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:02.405660 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:02.405660 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:02.405713 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:02.406015 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:02.406016 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:02.406016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:02.406152 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:02.406241 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:02.406628 (+ 387us) server_negotiation.cc:299] Negotiation successful
0929 10:55:02.406706 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 10:55:02.407045 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.395034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:36951 (local address 127.10.238.254:39293)
0929 10:55:02.395077 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:02.395079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:02.395345 (+ 266us) server_negotiation.cc:407] Connection header received
0929 10:55:02.396704 (+ 1359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:02.396705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:02.396719 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:02.396769 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:02.397695 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.398285 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.399258 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.399467 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.399555 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.404030 (+ 4475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:02.404032 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:02.404032 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:02.404040 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:02.405769 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:02.406184 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:02.406185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:02.406185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:02.406278 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:02.406463 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:02.406464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:02.406464 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:02.406595 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:02.406635 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:02.406831 (+ 196us) server_negotiation.cc:299] Negotiation successful
0929 10:55:02.406913 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 10:55:02.407188 18269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.395426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:59101 (local address 127.10.238.252:37593)
0929 10:55:02.395470 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:02.395473 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:02.395635 (+ 162us) server_negotiation.cc:407] Connection header received
0929 10:55:02.396827 (+ 1192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:02.396828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:02.396843 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:02.396895 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:02.397704 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.398496 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.399618 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.399943 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.399981 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.403756 (+ 3775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:02.403757 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:02.403757 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:02.403763 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:02.405982 (+ 2219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:02.406359 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:02.406360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:02.406361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:02.406412 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:02.406604 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:02.406605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:02.406605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:02.406736 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:02.406777 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:02.406984 (+ 207us) server_negotiation.cc:299] Negotiation successful
0929 10:55:02.407074 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 10:55:02.407559 18408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.395048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.253:45709 (local address 127.10.238.194:58025)
0929 10:55:02.395443 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:02.395452 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:02.396559 (+ 1107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:02.396850 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:02.396853 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:02.397382 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:02.397580 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:02.397598 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.398597 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.398597 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:02.399505 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:02.399507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.400031 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.400033 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:02.400097 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.400599 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:02.400604 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:02.402684 (+ 2080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:55:02.404759 (+ 2075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:02.404767 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:02.404770 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:02.405554 (+ 784us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:02.405784 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:02.405785 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:02.405785 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:02.405939 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:02.406291 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:55:02.406299 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:55:02.406461 (+ 162us) client_negotiation.cc:769] Sending connection context
0929 10:55:02.407047 (+ 586us) client_negotiation.cc:240] Negotiation successful
0929 10:55:02.407400 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":46,"thread_start_us":90,"threads_started":1}
I20260929 10:55:02.407531 18407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.395049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39293 (local address 127.10.238.194:36951)
0929 10:55:02.395212 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:02.395253 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:02.396547 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:02.396850 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:02.396854 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:02.397381 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:02.397579 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:02.397596 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.398390 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.398391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:02.399163 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:02.399165 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.399608 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.399609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:02.399741 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.400296 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:02.400306 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:02.403935 (+ 3629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:55:02.405883 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:02.405885 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:02.405885 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:02.406101 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:02.406345 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:02.406345 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:02.406346 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:02.406403 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:02.406675 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:55:02.406676 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:55:02.406716 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 10:55:02.407135 (+ 419us) client_negotiation.cc:240] Negotiation successful
0929 10:55:02.407292 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":96,"threads_started":1}
I20260929 10:55:02.407931 18406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.395332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.252:37593 (local address 127.10.238.194:59101)
0929 10:55:02.395577 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:02.395584 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:02.396565 (+ 981us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:02.396914 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:02.396915 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:02.397382 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:02.397580 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:02.397598 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.398597 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.398597 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:02.399505 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:02.399507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.400047 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.400047 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:02.400097 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.400681 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:02.400686 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:02.403635 (+ 2949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:55:02.406057 (+ 2422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:02.406057 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:02.406058 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:02.406282 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:02.406479 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:02.406480 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:02.406480 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:02.406534 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:02.406825 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:55:02.406826 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:55:02.406866 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 10:55:02.407305 (+ 439us) client_negotiation.cc:240] Negotiation successful
0929 10:55:02.407460 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":106,"thread_start_us":53,"threads_started":1}
I20260929 10:55:02.408301 18404 heartbeater.cc:347] Connected to a master server at 127.10.238.253:45709
I20260929 10:55:02.408396 18404 heartbeater.cc:464] Registering TS with master...
I20260929 10:55:02.408566 18402 heartbeater.cc:347] Connected to a master server at 127.10.238.252:37593
I20260929 10:55:02.408595 18404 heartbeater.cc:511] Master 127.10.238.253:45709 requested a full tablet report, sending...
I20260929 10:55:02.408627 18402 heartbeater.cc:464] Registering TS with master...
I20260929 10:55:02.408702 18402 heartbeater.cc:511] Master 127.10.238.252:37593 requested a full tablet report, sending...
I20260929 10:55:02.409085 18086 ts_manager.cc:194] Registered new tserver with Master: bb2eb90f68da49f7b9fb25af83502e1f (127.10.238.194:40983)
I20260929 10:55:02.409101 17899 ts_manager.cc:194] Registered new tserver with Master: bb2eb90f68da49f7b9fb25af83502e1f (127.10.238.194:40983)
I20260929 10:55:02.410142 18403 heartbeater.cc:347] Connected to a master server at 127.10.238.254:39293
I20260929 10:55:02.410212 18403 heartbeater.cc:464] Registering TS with master...
I20260929 10:55:02.410293 18403 heartbeater.cc:511] Master 127.10.238.254:39293 requested a full tablet report, sending...
I20260929 10:55:02.410429 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.195:45979
--local_ip_for_outbound_sockets=127.10.238.195
--tserver_master_addrs=127.10.238.254:39293
--webserver_port=40747
--webserver_interface=127.10.238.195
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.195.keytab
--principal=oryx/127.10.238.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:46719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.10.238.254:39293,127.10.238.253:45709
--tserver_master_addrs=127.10.238.254:39293,127.10.238.253:45709,127.10.238.252:37593 with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:55:02.410549 17300 ts_manager.cc:194] Re-registered known tserver with Master: bb2eb90f68da49f7b9fb25af83502e1f (127.10.238.194:40983)
I20260929 10:55:02.411051 17300 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.238.194@KRBTEST.COM'} at 127.10.238.194:36951
W20260929 10:55:02.493046 18409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:55:02.493314 18409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:55:02.493367 18409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:55:02.494525 18409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:55:02.494591 18409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:55:02.494637 18409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.195
I20260929 10:55:02.496430 18409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:46719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.195.keytab
--principal=oryx/127.10.238.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.195:45979
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.238.195
--webserver_port=40747
--tserver_master_addrs=127.10.238.254:39293,127.10.238.253:45709,127.10.238.252:37593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.195
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:55:02.496697 18409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:55:02.497007 18409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:55:02.501998 18414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:02.502027 18415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:02.502012 18417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:55:02.502187 18409 server_base.cc:1034] running on GCE node
I20260929 10:55:02.502417 18409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:55:02.502652 18409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:55:02.503818 18409 hybrid_clock.cc:648] HybridClock initialized: now 1790679302503809 us; error 50 us; skew 500 ppm
Sep 29 10:55:02 dist-test-slave-8nrt krb5kdc[17256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679302, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.238.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:55:02.506321 18409 init.cc:377] Logged in from keytab as oryx/127.10.238.195@KRBTEST.COM (short username oryx)
I20260929 10:55:02.506755 18409 webserver.cc:460] Webserver started at http://127.10.238.195:40747/ using document root <none> and password file <none>
I20260929 10:55:02.506942 18409 fs_manager.cc:359] Metadata directory not provided
I20260929 10:55:02.506990 18409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:55:02.508250 18409 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:55:02.508787 18424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:02.508985 18409 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:02.509053 18409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-2/wal
uuid: "99c69ddb2aee461092fbe82090413694"
format_stamp: "Formatted at 2026-09-29 10:55:01 on dist-test-slave-8nrt"
server_key: "b9a5e4a582e51d8608b150431979766e"
server_key_iv: "d5d7c6bb6917d57d997a553b30387173"
server_key_version: "encryptionkey@0"
I20260929 10:55:02.509419 18409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:55:02.515528 18409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:55:02.517134 18409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:55:02.517235 18409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:55:02.517467 18409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:55:02.517709 18409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:55:02.517736 18409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:02.517756 18409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:55:02.517768 18409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:02.523954 18409 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.195:45979
I20260929 10:55:02.524011 18537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.195:45979 every 8 connection(s)
I20260929 10:55:02.524327 18409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
I20260929 10:55:02.525431 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 18409
Sep 29 10:55:02 dist-test-slave-8nrt krb5kdc[17256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679302, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.238.195@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM
Sep 29 10:55:02 dist-test-slave-8nrt krb5kdc[17256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679302, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.238.195@KRBTEST.COM for oryx/127.10.238.253@KRBTEST.COM
I20260929 10:55:02.535607 18553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.527158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54912 (local address 127.10.238.252:37593)
0929 10:55:02.527413 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:02.527415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:02.527565 (+ 150us) server_negotiation.cc:407] Connection header received
0929 10:55:02.527592 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:02.527593 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:02.527607 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:02.527661 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:02.528297 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.529032 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.530034 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.530348 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.530401 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.532367 (+ 1966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:02.532368 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:02.532369 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:02.532375 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:02.534254 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:02.534652 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:02.534653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:02.534654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:02.534713 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:02.534911 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:02.534912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:02.534912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:02.535049 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:02.535144 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:02.535397 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 10:55:02.535479 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":52,"threads_started":1}
Sep 29 10:55:02 dist-test-slave-8nrt krb5kdc[17256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679302, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.238.195@KRBTEST.COM for oryx/127.10.238.252@KRBTEST.COM
I20260929 10:55:02.537585 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.524921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:56379 (local address 127.10.238.254:39293)
0929 10:55:02.524971 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:02.524974 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:02.526508 (+ 1534us) server_negotiation.cc:407] Connection header received
0929 10:55:02.527928 (+ 1420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:02.527929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:02.527940 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:02.527970 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:02.528925 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.529629 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.531083 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.531386 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.531428 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.534161 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:02.534163 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:02.534163 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:02.534170 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:02.535844 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:02.536376 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:02.536377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:02.536378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:02.536440 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:02.536772 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:02.536773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:02.536773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:02.536903 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:02.536949 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:02.537385 (+ 436us) server_negotiation.cc:299] Negotiation successful
0929 10:55:02.537466 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 10:55:02.538174 18543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.524920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39293 (local address 127.10.238.195:56379)
0929 10:55:02.526381 (+ 1461us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:02.526430 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:02.527807 (+ 1377us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:02.528107 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:02.528111 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:02.528546 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:02.528822 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:02.528828 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.529748 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.529749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:02.530982 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:02.530985 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.531512 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.531513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:02.531646 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.532167 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:02.532177 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:02.534047 (+ 1870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:55:02.535933 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:02.535940 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:02.535942 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:02.536266 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:02.536521 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:02.536522 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:02.536522 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:02.536684 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:02.537002 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:55:02.537009 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:55:02.537166 (+ 157us) client_negotiation.cc:769] Sending connection context
0929 10:55:02.537714 (+ 548us) client_negotiation.cc:240] Negotiation successful
0929 10:55:02.537901 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1419,"mutex_wait_us":2,"thread_start_us":269,"threads_started":1}
I20260929 10:55:02.538887 18539 heartbeater.cc:347] Connected to a master server at 127.10.238.254:39293
I20260929 10:55:02.538982 18539 heartbeater.cc:464] Registering TS with master...
I20260929 10:55:02.539170 18539 heartbeater.cc:511] Master 127.10.238.254:39293 requested a full tablet report, sending...
I20260929 10:55:02.539561 17300 ts_manager.cc:194] Re-registered known tserver with Master: 99c69ddb2aee461092fbe82090413694 (127.10.238.195:45979)
I20260929 10:55:02.540092 17300 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.238.195@KRBTEST.COM'} at 127.10.238.195:56379
I20260929 10:55:02.540820 18269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.525140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:35183 (local address 127.10.238.252:37593)
0929 10:55:02.525186 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:02.525188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:02.526510 (+ 1322us) server_negotiation.cc:407] Connection header received
0929 10:55:02.528078 (+ 1568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:02.528080 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:02.528089 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:02.528129 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:02.529404 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.530102 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.532518 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.533079 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.533148 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.537860 (+ 4712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:02.537861 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:02.537862 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:02.537868 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:02.539598 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:02.539976 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:02.539977 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:02.539978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:02.540032 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:02.540224 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:02.540225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:02.540225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:02.540372 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:02.540428 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:02.540640 (+ 212us) server_negotiation.cc:299] Negotiation successful
0929 10:55:02.540723 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 10:55:02.541249 18542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.524920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.252:37593 (local address 127.10.238.195:35183)
0929 10:55:02.526412 (+ 1492us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:02.526431 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:02.528003 (+ 1572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:02.528179 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:02.528179 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:02.529244 (+ 1065us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:02.529328 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:02.529330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.530194 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.530195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:02.532425 (+ 2230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:02.532427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.533464 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.533465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:02.533534 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.534595 (+ 1061us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:02.534600 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:02.537768 (+ 3168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:55:02.539691 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:02.539692 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:02.539692 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:02.539902 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:02.540101 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:02.540102 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:02.540102 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:02.540156 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:02.540453 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:55:02.540454 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:55:02.540493 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 10:55:02.540564 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 10:55:02.541121 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1467,"mutex_wait_us":440,"thread_start_us":281,"threads_started":1}
I20260929 10:55:02.541445 17950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.526693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48006 (local address 127.10.238.253:45709)
0929 10:55:02.526755 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:02.526757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:02.527103 (+ 346us) server_negotiation.cc:407] Connection header received
0929 10:55:02.527202 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:02.527203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:02.527215 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:02.527247 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:02.529817 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.530481 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.532943 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.533246 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.533344 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.537131 (+ 3787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:02.537132 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:02.537132 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:02.537140 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:02.538634 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:02.540094 (+ 1460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:02.540095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:02.540095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:02.540146 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:02.540349 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:02.540350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:02.540351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:02.540478 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:02.540551 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:02.541241 (+ 690us) server_negotiation.cc:299] Negotiation successful
0929 10:55:02.541346 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 10:55:02.541584 18538 heartbeater.cc:347] Connected to a master server at 127.10.238.252:37593
I20260929 10:55:02.541626 18538 heartbeater.cc:464] Registering TS with master...
I20260929 10:55:02.541695 18538 heartbeater.cc:511] Master 127.10.238.252:37593 requested a full tablet report, sending...
I20260929 10:55:02.541877 18549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.526602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54828 (local address 127.10.238.254:39293)
0929 10:55:02.526777 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:02.526778 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:02.526919 (+ 141us) server_negotiation.cc:407] Connection header received
0929 10:55:02.527022 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:02.527024 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:02.527039 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:02.527115 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:02.528368 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.529099 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.535363 (+ 6264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.535689 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.535767 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.538879 (+ 3112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:02.538880 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:02.538881 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:02.538888 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:02.540550 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:02.540923 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:02.540924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:02.540924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:02.540992 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:02.541187 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:02.541188 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:02.541188 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:02.541347 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:02.541442 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:02.541678 (+ 236us) server_negotiation.cc:299] Negotiation successful
0929 10:55:02.541766 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":81,"threads_started":1}
I20260929 10:55:02.541965 18086 ts_manager.cc:194] Registered new tserver with Master: 99c69ddb2aee461092fbe82090413694 (127.10.238.195:45979)
I20260929 10:55:02.542073 18548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.526456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.253:45709 (local address 127.10.238.195:53095)
0929 10:55:02.526573 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:02.526583 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:02.527810 (+ 1227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:02.528315 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:02.528316 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:02.528558 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:02.528822 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:02.528828 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.531539 (+ 2711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.531540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:02.532422 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:02.532424 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.533678 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.533679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:02.533745 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.534105 (+ 360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:02.534110 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:02.535515 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:55:02.540743 (+ 5228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:02.540743 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:02.540744 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:02.540964 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:02.541189 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:02.541190 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:02.541190 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:02.541295 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:02.541576 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:55:02.541577 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:55:02.541614 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 10:55:02.541669 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 10:55:02.541773 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":20,"thread_start_us":62,"threads_started":1}
I20260929 10:55:02.542402 18551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.526820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:53095 (local address 127.10.238.253:45709)
0929 10:55:02.526918 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:02.526920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:02.526928 (+ 8us) server_negotiation.cc:407] Connection header received
0929 10:55:02.528210 (+ 1282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:02.528211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:02.528223 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:02.528488 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:02.528929 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.529706 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.532512 (+ 2806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.532842 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.533898 (+ 1056us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.535602 (+ 1704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:02.535603 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:02.535603 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:02.535609 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:02.539766 (+ 4157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:02.541047 (+ 1281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:02.541048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:02.541048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:02.541103 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:02.541367 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:02.541368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:02.541368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:02.541494 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:02.542165 (+ 671us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:02.542210 (+ 45us) server_negotiation.cc:299] Negotiation successful
0929 10:55:02.542296 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":63,"threads_started":1}
I20260929 10:55:02.542840 18540 heartbeater.cc:347] Connected to a master server at 127.10.238.253:45709
I20260929 10:55:02.542892 18540 heartbeater.cc:464] Registering TS with master...
I20260929 10:55:02.542954 18540 heartbeater.cc:511] Master 127.10.238.253:45709 requested a full tablet report, sending...
I20260929 10:55:02.543246 17899 ts_manager.cc:194] Registered new tserver with Master: 99c69ddb2aee461092fbe82090413694 (127.10.238.195:45979)
I20260929 10:55:02.543883 17300 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54828:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:55:02.545433 17300 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:55:02.552911 18563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.549389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33278 (local address 127.10.238.193:42961)
0929 10:55:02.549582 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:02.549615 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:02.549625 (+ 10us) server_negotiation.cc:407] Connection header received
0929 10:55:02.549667 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:02.549668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:02.549684 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:02.549918 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:55:02.549969 (+ 51us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.550608 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.551978 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.552567 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.552687 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.552755 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:02.552791 (+ 36us) server_negotiation.cc:299] Negotiation successful
0929 10:55:02.552833 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":95,"threads_started":1}
I20260929 10:55:02.552976 17859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.549017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.195:45979 (local address 127.0.0.1:41756)
0929 10:55:02.549203 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:02.549213 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:02.549273 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:02.549560 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:02.549561 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:02.549563 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:55:02.549634 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:02.549636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.550905 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.550906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:02.551950 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:02.551951 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.552749 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.552750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:02.552809 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.552813 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 10:55:02.552843 (+ 30us) client_negotiation.cc:240] Negotiation successful
0929 10:55:02.552880 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172}
I20260929 10:55:02.552876 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.549044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:42961 (local address 127.0.0.1:33278)
0929 10:55:02.549416 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:02.549424 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:02.549458 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:02.549786 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:02.549787 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:02.549789 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:55:02.549847 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:02.549848 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.550694 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.550695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:02.551895 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:02.551897 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.552650 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.552651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:02.552727 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.552731 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 10:55:02.552770 (+ 39us) client_negotiation.cc:240] Negotiation successful
0929 10:55:02.552809 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":23,"thread_start_us":72,"threads_started":1}
I20260929 10:55:02.553382 18560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.549106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.194:40983 (local address 127.0.0.1:46756)
0929 10:55:02.549694 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:02.549705 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:02.549765 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:02.550019 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:02.550020 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:02.550022 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:55:02.550070 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:02.550072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.551180 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.551181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:02.552141 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:02.552144 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.553126 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.553127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:02.553195 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.553198 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 10:55:02.553246 (+ 48us) client_negotiation.cc:240] Negotiation successful
0929 10:55:02.553300 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":65,"threads_started":1}
I20260929 10:55:02.553449 18562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.549207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46756 (local address 127.10.238.194:40983)
0929 10:55:02.549583 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:02.549616 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:02.549824 (+ 208us) server_negotiation.cc:407] Connection header received
0929 10:55:02.549866 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:02.549869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:02.549884 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:02.549952 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:55:02.550142 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.551099 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.552226 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.552950 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.553036 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.553094 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:02.553295 (+ 201us) server_negotiation.cc:299] Negotiation successful
0929 10:55:02.553342 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":122,"threads_started":1}
I20260929 10:55:02.553838 18199 tablet_service.cc:1467] Processing CreateTablet for tablet 0222627b936547e0951837de3e914df4 (DEFAULT_TABLE table=test-table [id=0580daf902c94c5a93b5d9cd6001c6c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:55:02.554171 18199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0222627b936547e0951837de3e914df4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:55:02.554097 18336 tablet_service.cc:1467] Processing CreateTablet for tablet 0222627b936547e0951837de3e914df4 (DEFAULT_TABLE table=test-table [id=0580daf902c94c5a93b5d9cd6001c6c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:55:02.554373 18336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0222627b936547e0951837de3e914df4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:55:02.556875 18564 tablet_bootstrap.cc:492] T 0222627b936547e0951837de3e914df4 P 2b18011dc9b24cf587c3bb2237afc7e6: Bootstrap starting.
I20260929 10:55:02.557358 18561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.549157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41756 (local address 127.10.238.195:45979)
0929 10:55:02.549375 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:02.549379 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:02.549389 (+ 10us) server_negotiation.cc:407] Connection header received
0929 10:55:02.549424 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:02.549426 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:02.549441 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:02.549714 (+ 273us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:55:02.549763 (+ 49us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.550825 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.552029 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.552634 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.552943 (+ 309us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.557176 (+ 4233us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:02.557199 (+ 23us) server_negotiation.cc:299] Negotiation successful
0929 10:55:02.557230 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":146,"threads_started":1}
I20260929 10:55:02.557582 18564 tablet_bootstrap.cc:654] T 0222627b936547e0951837de3e914df4 P 2b18011dc9b24cf587c3bb2237afc7e6: Neither blocks nor log segments found. Creating new log.
I20260929 10:55:02.557925 18564 log.cc:824] T 0222627b936547e0951837de3e914df4 P 2b18011dc9b24cf587c3bb2237afc7e6: Log is configured to *not* fsync() on all Append() calls
I20260929 10:55:02.558027 18472 tablet_service.cc:1467] Processing CreateTablet for tablet 0222627b936547e0951837de3e914df4 (DEFAULT_TABLE table=test-table [id=0580daf902c94c5a93b5d9cd6001c6c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:55:02.558256 18565 tablet_bootstrap.cc:492] T 0222627b936547e0951837de3e914df4 P bb2eb90f68da49f7b9fb25af83502e1f: Bootstrap starting.
I20260929 10:55:02.558355 18472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0222627b936547e0951837de3e914df4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:55:02.558787 18564 tablet_bootstrap.cc:492] T 0222627b936547e0951837de3e914df4 P 2b18011dc9b24cf587c3bb2237afc7e6: No bootstrap required, opened a new log
I20260929 10:55:02.558866 18564 ts_tablet_manager.cc:1397] T 0222627b936547e0951837de3e914df4 P 2b18011dc9b24cf587c3bb2237afc7e6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 10:55:02.558894 18565 tablet_bootstrap.cc:654] T 0222627b936547e0951837de3e914df4 P bb2eb90f68da49f7b9fb25af83502e1f: Neither blocks nor log segments found. Creating new log.
I20260929 10:55:02.559185 18565 log.cc:824] T 0222627b936547e0951837de3e914df4 P bb2eb90f68da49f7b9fb25af83502e1f: Log is configured to *not* fsync() on all Append() calls
I20260929 10:55:02.560070 18565 tablet_bootstrap.cc:492] T 0222627b936547e0951837de3e914df4 P bb2eb90f68da49f7b9fb25af83502e1f: No bootstrap required, opened a new log
I20260929 10:55:02.560142 18565 ts_tablet_manager.cc:1397] T 0222627b936547e0951837de3e914df4 P bb2eb90f68da49f7b9fb25af83502e1f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 10:55:02.561417 18568 tablet_bootstrap.cc:492] T 0222627b936547e0951837de3e914df4 P 99c69ddb2aee461092fbe82090413694: Bootstrap starting.
I20260929 10:55:02.561600 18564 raft_consensus.cc:348] T 0222627b936547e0951837de3e914df4 P 2b18011dc9b24cf587c3bb2237afc7e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c69ddb2aee461092fbe82090413694" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 45979 } } peers { permanent_uuid: "2b18011dc9b24cf587c3bb2237afc7e6" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 42961 } } peers { permanent_uuid: "bb2eb90f68da49f7b9fb25af83502e1f" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40983 } }
I20260929 10:55:02.561771 18564 raft_consensus.cc:374] T 0222627b936547e0951837de3e914df4 P 2b18011dc9b24cf587c3bb2237afc7e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:55:02.561815 18564 raft_consensus.cc:729] T 0222627b936547e0951837de3e914df4 P 2b18011dc9b24cf587c3bb2237afc7e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b18011dc9b24cf587c3bb2237afc7e6, State: Initialized, Role: FOLLOWER
I20260929 10:55:02.561944 18564 consensus_queue.cc:260] T 0222627b936547e0951837de3e914df4 P 2b18011dc9b24cf587c3bb2237afc7e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c69ddb2aee461092fbe82090413694" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 45979 } } peers { permanent_uuid: "2b18011dc9b24cf587c3bb2237afc7e6" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 42961 } } peers { permanent_uuid: "bb2eb90f68da49f7b9fb25af83502e1f" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40983 } }
I20260929 10:55:02.562152 18568 tablet_bootstrap.cc:654] T 0222627b936547e0951837de3e914df4 P 99c69ddb2aee461092fbe82090413694: Neither blocks nor log segments found. Creating new log.
I20260929 10:55:02.562242 18564 ts_tablet_manager.cc:1428] T 0222627b936547e0951837de3e914df4 P 2b18011dc9b24cf587c3bb2237afc7e6: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260929 10:55:02.562363 18266 heartbeater.cc:503] Master 127.10.238.254:39293 was elected leader, sending a full tablet report...
I20260929 10:55:02.562467 18568 log.cc:824] T 0222627b936547e0951837de3e914df4 P 99c69ddb2aee461092fbe82090413694: Log is configured to *not* fsync() on all Append() calls
I20260929 10:55:02.563037 18565 raft_consensus.cc:348] T 0222627b936547e0951837de3e914df4 P bb2eb90f68da49f7b9fb25af83502e1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c69ddb2aee461092fbe82090413694" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 45979 } } peers { permanent_uuid: "2b18011dc9b24cf587c3bb2237afc7e6" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 42961 } } peers { permanent_uuid: "bb2eb90f68da49f7b9fb25af83502e1f" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40983 } }
I20260929 10:55:02.563221 18565 raft_consensus.cc:374] T 0222627b936547e0951837de3e914df4 P bb2eb90f68da49f7b9fb25af83502e1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:55:02.563272 18565 raft_consensus.cc:729] T 0222627b936547e0951837de3e914df4 P bb2eb90f68da49f7b9fb25af83502e1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb2eb90f68da49f7b9fb25af83502e1f, State: Initialized, Role: FOLLOWER
I20260929 10:55:02.563582 18565 consensus_queue.cc:260] T 0222627b936547e0951837de3e914df4 P bb2eb90f68da49f7b9fb25af83502e1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c69ddb2aee461092fbe82090413694" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 45979 } } peers { permanent_uuid: "2b18011dc9b24cf587c3bb2237afc7e6" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 42961 } } peers { permanent_uuid: "bb2eb90f68da49f7b9fb25af83502e1f" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40983 } }
I20260929 10:55:02.563858 18565 ts_tablet_manager.cc:1428] T 0222627b936547e0951837de3e914df4 P bb2eb90f68da49f7b9fb25af83502e1f: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260929 10:55:02.563884 18403 heartbeater.cc:503] Master 127.10.238.254:39293 was elected leader, sending a full tablet report...
I20260929 10:55:02.564554 18568 tablet_bootstrap.cc:492] T 0222627b936547e0951837de3e914df4 P 99c69ddb2aee461092fbe82090413694: No bootstrap required, opened a new log
I20260929 10:55:02.564623 18568 ts_tablet_manager.cc:1397] T 0222627b936547e0951837de3e914df4 P 99c69ddb2aee461092fbe82090413694: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 10:55:02.566174 18568 raft_consensus.cc:348] T 0222627b936547e0951837de3e914df4 P 99c69ddb2aee461092fbe82090413694 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c69ddb2aee461092fbe82090413694" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 45979 } } peers { permanent_uuid: "2b18011dc9b24cf587c3bb2237afc7e6" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 42961 } } peers { permanent_uuid: "bb2eb90f68da49f7b9fb25af83502e1f" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40983 } }
I20260929 10:55:02.566295 18568 raft_consensus.cc:374] T 0222627b936547e0951837de3e914df4 P 99c69ddb2aee461092fbe82090413694 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:55:02.566316 18568 raft_consensus.cc:729] T 0222627b936547e0951837de3e914df4 P 99c69ddb2aee461092fbe82090413694 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99c69ddb2aee461092fbe82090413694, State: Initialized, Role: FOLLOWER
I20260929 10:55:02.566473 18568 consensus_queue.cc:260] T 0222627b936547e0951837de3e914df4 P 99c69ddb2aee461092fbe82090413694 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c69ddb2aee461092fbe82090413694" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 45979 } } peers { permanent_uuid: "2b18011dc9b24cf587c3bb2237afc7e6" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 42961 } } peers { permanent_uuid: "bb2eb90f68da49f7b9fb25af83502e1f" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40983 } }
I20260929 10:55:02.566766 18568 ts_tablet_manager.cc:1428] T 0222627b936547e0951837de3e914df4 P 99c69ddb2aee461092fbe82090413694: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 10:55:02.566788 18539 heartbeater.cc:503] Master 127.10.238.254:39293 was elected leader, sending a full tablet report...
I20260929 10:55:02.596040 18560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.593211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.252:37593 (local address 127.0.0.1:54914)
0929 10:55:02.593289 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:02.593301 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:02.593369 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:02.593530 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:02.593531 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:02.593533 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:55:02.593590 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:02.593596 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.594376 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.594377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:02.595171 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:02.595173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.595795 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.595796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:02.595865 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.595869 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 10:55:02.595922 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 10:55:02.595966 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 10:55:02.596043 18269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.593310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54914 (local address 127.10.238.252:37593)
0929 10:55:02.593382 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:02.593384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:02.593395 (+ 11us) server_negotiation.cc:407] Connection header received
0929 10:55:02.593424 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:02.593425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:02.593452 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:02.593528 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:55:02.593664 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.594291 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.595252 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.595707 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.595792 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.595884 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:02.595932 (+ 48us) server_negotiation.cc:299] Negotiation successful
0929 10:55:02.595966 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 10:55:02.596560 18096 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260929 10:55:02.597924 18125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82fdb61571ad46209f5fe7dca6fae4bf. Latest consensus state: current_term: 1 leader_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } } peers { permanent_uuid: "59092f98066b4395b785a2698bf678c7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45709 } attrs { promote: false } } peers { permanent_uuid: "952082091a5949e68504685e56cd3c94" member_type: NON_VOTER last_known_addr { host: "127.10.238.252" port: 37593 } attrs { promote: true } } }
I20260929 10:55:02.598023 18125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94 [sys.catalog]: This master's current role is: LEARNER
W20260929 10:55:02.645318 18405 tablet.cc:2369] T 0222627b936547e0951837de3e914df4 P bb2eb90f68da49f7b9fb25af83502e1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 10:55:02.764878 18268 tablet.cc:2369] T 0222627b936547e0951837de3e914df4 P 2b18011dc9b24cf587c3bb2237afc7e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 10:55:02.775419 18541 tablet.cc:2369] T 0222627b936547e0951837de3e914df4 P 99c69ddb2aee461092fbe82090413694: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:55:02.852180 18569 raft_consensus.cc:482] T 0222627b936547e0951837de3e914df4 P 2b18011dc9b24cf587c3bb2237afc7e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:55:02.852344 18569 raft_consensus.cc:504] T 0222627b936547e0951837de3e914df4 P 2b18011dc9b24cf587c3bb2237afc7e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c69ddb2aee461092fbe82090413694" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 45979 } } peers { permanent_uuid: "2b18011dc9b24cf587c3bb2237afc7e6" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 42961 } } peers { permanent_uuid: "bb2eb90f68da49f7b9fb25af83502e1f" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40983 } }
I20260929 10:55:02.852730 18569 leader_election.cc:288] T 0222627b936547e0951837de3e914df4 P 2b18011dc9b24cf587c3bb2237afc7e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99c69ddb2aee461092fbe82090413694 (127.10.238.195:45979), bb2eb90f68da49f7b9fb25af83502e1f (127.10.238.194:40983)
I20260929 10:55:02.856218 18562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.853366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:40043 (local address 127.10.238.194:40983)
0929 10:55:02.853440 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:02.853442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:02.853454 (+ 12us) server_negotiation.cc:407] Connection header received
0929 10:55:02.853493 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:02.853494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:02.853503 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:02.853552 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:55:02.853730 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.854238 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.855340 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.855919 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.855985 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.855992 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:02.856135 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 10:55:02.856166 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 10:55:02.856246 18577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.853238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.194:40983 (local address 127.10.238.193:40043)
0929 10:55:02.853349 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:02.853360 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:02.853437 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:02.853563 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:02.853563 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:02.853565 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:55:02.853631 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:02.853632 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.854314 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.854315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:02.855259 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:02.855261 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.855992 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.855993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:02.856060 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.856064 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 10:55:02.856121 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 10:55:02.856161 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":70,"threads_started":1}
I20260929 10:55:02.856768 18576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.853018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.195:45979 (local address 127.10.238.193:60907)
0929 10:55:02.853171 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:02.853183 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:02.853250 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:02.853503 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:02.853504 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:02.853507 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:55:02.853599 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:02.853602 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.854904 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.854905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:02.855833 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:02.855835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.856560 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.856561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:02.856633 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.856637 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 10:55:02.856686 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 10:55:02.856716 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":98,"threads_started":1}
I20260929 10:55:02.856719 18356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0222627b936547e0951837de3e914df4" candidate_uuid: "2b18011dc9b24cf587c3bb2237afc7e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb2eb90f68da49f7b9fb25af83502e1f" is_pre_election: true
I20260929 10:55:02.856817 18561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.853095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:60907 (local address 127.10.238.195:45979)
0929 10:55:02.853165 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:02.853167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:02.853251 (+ 84us) server_negotiation.cc:407] Connection header received
0929 10:55:02.853349 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:02.853353 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:02.853368 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:02.853420 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:55:02.853688 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.854813 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.855922 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.856474 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.856571 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.856579 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:02.856716 (+ 137us) server_negotiation.cc:299] Negotiation successful
0929 10:55:02.856758 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 10:55:02.856890 18356 raft_consensus.cc:2454] T 0222627b936547e0951837de3e914df4 P bb2eb90f68da49f7b9fb25af83502e1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b18011dc9b24cf587c3bb2237afc7e6 in term 0.
I20260929 10:55:02.857142 18152 leader_election.cc:302] T 0222627b936547e0951837de3e914df4 P 2b18011dc9b24cf587c3bb2237afc7e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b18011dc9b24cf587c3bb2237afc7e6, bb2eb90f68da49f7b9fb25af83502e1f; no voters:
I20260929 10:55:02.857219 18492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0222627b936547e0951837de3e914df4" candidate_uuid: "2b18011dc9b24cf587c3bb2237afc7e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99c69ddb2aee461092fbe82090413694" is_pre_election: true
I20260929 10:55:02.857287 18569 raft_consensus.cc:2781] T 0222627b936547e0951837de3e914df4 P 2b18011dc9b24cf587c3bb2237afc7e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:55:02.857344 18569 raft_consensus.cc:482] T 0222627b936547e0951837de3e914df4 P 2b18011dc9b24cf587c3bb2237afc7e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:55:02.857398 18569 raft_consensus.cc:3037] T 0222627b936547e0951837de3e914df4 P 2b18011dc9b24cf587c3bb2237afc7e6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:55:02.857398 18492 raft_consensus.cc:2454] T 0222627b936547e0951837de3e914df4 P 99c69ddb2aee461092fbe82090413694 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b18011dc9b24cf587c3bb2237afc7e6 in term 0.
I20260929 10:55:02.858132 18569 raft_consensus.cc:504] T 0222627b936547e0951837de3e914df4 P 2b18011dc9b24cf587c3bb2237afc7e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c69ddb2aee461092fbe82090413694" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 45979 } } peers { permanent_uuid: "2b18011dc9b24cf587c3bb2237afc7e6" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 42961 } } peers { permanent_uuid: "bb2eb90f68da49f7b9fb25af83502e1f" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40983 } }
I20260929 10:55:02.858265 18569 leader_election.cc:288] T 0222627b936547e0951837de3e914df4 P 2b18011dc9b24cf587c3bb2237afc7e6 [CANDIDATE]: Term 1 election: Requested vote from peers 99c69ddb2aee461092fbe82090413694 (127.10.238.195:45979), bb2eb90f68da49f7b9fb25af83502e1f (127.10.238.194:40983)
I20260929 10:55:02.858435 18492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0222627b936547e0951837de3e914df4" candidate_uuid: "2b18011dc9b24cf587c3bb2237afc7e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99c69ddb2aee461092fbe82090413694"
I20260929 10:55:02.858517 18356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0222627b936547e0951837de3e914df4" candidate_uuid: "2b18011dc9b24cf587c3bb2237afc7e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb2eb90f68da49f7b9fb25af83502e1f"
I20260929 10:55:02.858525 18492 raft_consensus.cc:3037] T 0222627b936547e0951837de3e914df4 P 99c69ddb2aee461092fbe82090413694 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:55:02.858623 18356 raft_consensus.cc:3037] T 0222627b936547e0951837de3e914df4 P bb2eb90f68da49f7b9fb25af83502e1f [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:55:02.859268 18492 raft_consensus.cc:2454] T 0222627b936547e0951837de3e914df4 P 99c69ddb2aee461092fbe82090413694 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b18011dc9b24cf587c3bb2237afc7e6 in term 1.
I20260929 10:55:02.859378 18356 raft_consensus.cc:2454] T 0222627b936547e0951837de3e914df4 P bb2eb90f68da49f7b9fb25af83502e1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b18011dc9b24cf587c3bb2237afc7e6 in term 1.
I20260929 10:55:02.859447 18152 leader_election.cc:302] T 0222627b936547e0951837de3e914df4 P 2b18011dc9b24cf587c3bb2237afc7e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b18011dc9b24cf587c3bb2237afc7e6, 99c69ddb2aee461092fbe82090413694; no voters:
I20260929 10:55:02.859589 18569 raft_consensus.cc:2781] T 0222627b936547e0951837de3e914df4 P 2b18011dc9b24cf587c3bb2237afc7e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:55:02.859723 18569 raft_consensus.cc:686] T 0222627b936547e0951837de3e914df4 P 2b18011dc9b24cf587c3bb2237afc7e6 [term 1 LEADER]: Becoming Leader. State: Replica: 2b18011dc9b24cf587c3bb2237afc7e6, State: Running, Role: LEADER
I20260929 10:55:02.859813 18569 consensus_queue.cc:237] T 0222627b936547e0951837de3e914df4 P 2b18011dc9b24cf587c3bb2237afc7e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c69ddb2aee461092fbe82090413694" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 45979 } } peers { permanent_uuid: "2b18011dc9b24cf587c3bb2237afc7e6" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 42961 } } peers { permanent_uuid: "bb2eb90f68da49f7b9fb25af83502e1f" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40983 } }
I20260929 10:55:02.860659 17300 catalog_manager.cc:5495] T 0222627b936547e0951837de3e914df4 P 2b18011dc9b24cf587c3bb2237afc7e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b18011dc9b24cf587c3bb2237afc7e6 (127.10.238.193). New cstate: current_term: 1 leader_uuid: "2b18011dc9b24cf587c3bb2237afc7e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c69ddb2aee461092fbe82090413694" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b18011dc9b24cf587c3bb2237afc7e6" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 42961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb2eb90f68da49f7b9fb25af83502e1f" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40983 } health_report { overall_health: UNKNOWN } } }
I20260929 10:55:02.861208 17861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } } peers { permanent_uuid: "59092f98066b4395b785a2698bf678c7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45709 } attrs { promote: false } } peers { permanent_uuid: "952082091a5949e68504685e56cd3c94" member_type: NON_VOTER last_known_addr { host: "127.10.238.252" port: 37593 } attrs { promote: true } } }
I20260929 10:55:02.861320 17861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf [sys.catalog]: This master's current role is: LEADER
I20260929 10:55:02.862035 18050 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf: attempting to promote NON_VOTER 952082091a5949e68504685e56cd3c94 to VOTER
I20260929 10:55:02.862169 18050 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } } peers { permanent_uuid: "59092f98066b4395b785a2698bf678c7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45709 } attrs { promote: false } } peers { permanent_uuid: "952082091a5949e68504685e56cd3c94" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 37593 } attrs { promote: false } }
I20260929 10:55:02.862530 17909 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 [term 1 FOLLOWER]: Refusing update from remote peer 82fdb61571ad46209f5fe7dca6fae4bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260929 10:55:02.862576 18096 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94 [term 1 LEARNER]: Refusing update from remote peer 82fdb61571ad46209f5fe7dca6fae4bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260929 10:55:02.862756 18050 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "952082091a5949e68504685e56cd3c94" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 37593 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260929 10:55:02.862857 18049 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "59092f98066b4395b785a2698bf678c7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45709 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260929 10:55:02.863379 18050 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 952082091a5949e68504685e56cd3c94 (127.10.238.252) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } } peers { permanent_uuid: "59092f98066b4395b785a2698bf678c7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45709 } attrs { promote: false } } peers { permanent_uuid: "952082091a5949e68504685e56cd3c94" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 37593 } attrs { promote: false } } }
I20260929 10:55:02.863497 18096 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 952082091a5949e68504685e56cd3c94 (127.10.238.252) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } } peers { permanent_uuid: "59092f98066b4395b785a2698bf678c7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45709 } attrs { promote: false } } peers { permanent_uuid: "952082091a5949e68504685e56cd3c94" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 37593 } attrs { promote: false } } }
I20260929 10:55:02.864111 18050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } } peers { permanent_uuid: "59092f98066b4395b785a2698bf678c7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45709 } attrs { promote: false } } peers { permanent_uuid: "952082091a5949e68504685e56cd3c94" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 37593 } attrs { promote: false } } }
I20260929 10:55:02.864179 18050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf [sys.catalog]: This master's current role is: LEADER
I20260929 10:55:02.864161 18049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } } peers { permanent_uuid: "59092f98066b4395b785a2698bf678c7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45709 } attrs { promote: false } } peers { permanent_uuid: "952082091a5949e68504685e56cd3c94" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 37593 } attrs { promote: false } } }
I20260929 10:55:02.864233 18049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf [sys.catalog]: This master's current role is: LEADER
I20260929 10:55:02.864420 18049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } } peers { permanent_uuid: "59092f98066b4395b785a2698bf678c7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45709 } attrs { promote: false } } peers { permanent_uuid: "952082091a5949e68504685e56cd3c94" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 37593 } attrs { promote: false } } }
I20260929 10:55:02.864516 18049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf [sys.catalog]: This master's current role is: LEADER
I20260929 10:55:02.864488 18125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } } peers { permanent_uuid: "59092f98066b4395b785a2698bf678c7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45709 } attrs { promote: false } } peers { permanent_uuid: "952082091a5949e68504685e56cd3c94" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 37593 } attrs { promote: false } } }
I20260929 10:55:02.864600 18125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:55:02.864646 17909 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 952082091a5949e68504685e56cd3c94 (127.10.238.252) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } } peers { permanent_uuid: "59092f98066b4395b785a2698bf678c7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45709 } attrs { promote: false } } peers { permanent_uuid: "952082091a5949e68504685e56cd3c94" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 37593 } attrs { promote: false } } }
I20260929 10:55:02.867125 18581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } } peers { permanent_uuid: "59092f98066b4395b785a2698bf678c7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45709 } attrs { promote: false } } peers { permanent_uuid: "952082091a5949e68504685e56cd3c94" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 37593 } attrs { promote: false } } }
I20260929 10:55:02.867197 18581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:55:02.903992 18563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.901820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33286 (local address 127.10.238.193:42961)
0929 10:55:02.901878 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:02.901880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:02.901888 (+ 8us) server_negotiation.cc:407] Connection header received
0929 10:55:02.901940 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:02.901941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:02.901958 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:02.902016 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:55:02.902126 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.902586 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.903279 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.903470 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.903519 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.903652 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:55:02.903763 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:55:02.903822 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:02.903858 (+ 36us) server_negotiation.cc:299] Negotiation successful
0929 10:55:02.903877 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 10:55:02.905853 18356 raft_consensus.cc:1261] T 0222627b936547e0951837de3e914df4 P bb2eb90f68da49f7b9fb25af83502e1f [term 1 FOLLOWER]: Refusing update from remote peer 2b18011dc9b24cf587c3bb2237afc7e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:55:02.905890 18492 raft_consensus.cc:1261] T 0222627b936547e0951837de3e914df4 P 99c69ddb2aee461092fbe82090413694 [term 1 FOLLOWER]: Refusing update from remote peer 2b18011dc9b24cf587c3bb2237afc7e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:55:02.906195 18578 consensus_queue.cc:1035] T 0222627b936547e0951837de3e914df4 P 2b18011dc9b24cf587c3bb2237afc7e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb2eb90f68da49f7b9fb25af83502e1f" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:55:02.906353 18569 consensus_queue.cc:1035] T 0222627b936547e0951837de3e914df4 P 2b18011dc9b24cf587c3bb2237afc7e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99c69ddb2aee461092fbe82090413694" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:55:02.909440 18584 mvcc.cc:205] Tried to move back new op lower bound from 7334622424699445248 to 7334622424514306048. Current Snapshot: MvccSnapshot[applied={T|T < 7334622424699445248}]
I20260929 10:55:02.911047 17300 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:55:02.911144 17300 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:55:02.912669 17300 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 82fdb61571ad46209f5fe7dca6fae4bf: Sending DeleteTablet for 3 replicas of tablet 0222627b936547e0951837de3e914df4
I20260929 10:55:02.913004 18336 tablet_service.cc:1514] Processing DeleteTablet for tablet 0222627b936547e0951837de3e914df4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:55:02 UTC) from {username='oryx', principal='oryx/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:46756
I20260929 10:55:02.913030 18472 tablet_service.cc:1514] Processing DeleteTablet for tablet 0222627b936547e0951837de3e914df4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:55:02 UTC) from {username='oryx', principal='oryx/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:41756
I20260929 10:55:02.913215 18594 tablet_replica.cc:330] T 0222627b936547e0951837de3e914df4 P 99c69ddb2aee461092fbe82090413694: stopping tablet replica
I20260929 10:55:02.913326 18594 raft_consensus.cc:2229] T 0222627b936547e0951837de3e914df4 P 99c69ddb2aee461092fbe82090413694 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:55:02.913424 18594 raft_consensus.cc:2258] T 0222627b936547e0951837de3e914df4 P 99c69ddb2aee461092fbe82090413694 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:55:02.913966 18594 ts_tablet_manager.cc:1905] T 0222627b936547e0951837de3e914df4 P 99c69ddb2aee461092fbe82090413694: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:55:02.914544 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 17269
I20260929 10:55:02.915792 18594 ts_tablet_manager.cc:1918] T 0222627b936547e0951837de3e914df4 P 99c69ddb2aee461092fbe82090413694: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:55:02.915870 18594 log.cc:1196] T 0222627b936547e0951837de3e914df4 P 99c69ddb2aee461092fbe82090413694: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-2/wal/wals/0222627b936547e0951837de3e914df4
I20260929 10:55:02.916144 18594 ts_tablet_manager.cc:1939] T 0222627b936547e0951837de3e914df4 P 99c69ddb2aee461092fbe82090413694: Deleting consensus metadata
I20260929 10:55:02.916301 18593 tablet_replica.cc:330] T 0222627b936547e0951837de3e914df4 P bb2eb90f68da49f7b9fb25af83502e1f: stopping tablet replica
I20260929 10:55:02.916457 18593 raft_consensus.cc:2229] T 0222627b936547e0951837de3e914df4 P bb2eb90f68da49f7b9fb25af83502e1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:55:02.916636 18593 raft_consensus.cc:2258] T 0222627b936547e0951837de3e914df4 P bb2eb90f68da49f7b9fb25af83502e1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:55:02.916661 18199 tablet_service.cc:1514] Processing DeleteTablet for tablet 0222627b936547e0951837de3e914df4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:55:02 UTC) from {username='oryx', principal='oryx/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:33278
I20260929 10:55:02.916874 18596 tablet_replica.cc:330] T 0222627b936547e0951837de3e914df4 P 2b18011dc9b24cf587c3bb2237afc7e6: stopping tablet replica
I20260929 10:55:02.917007 18596 raft_consensus.cc:2229] T 0222627b936547e0951837de3e914df4 P 2b18011dc9b24cf587c3bb2237afc7e6 [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:55:02.917124 18593 ts_tablet_manager.cc:1905] T 0222627b936547e0951837de3e914df4 P bb2eb90f68da49f7b9fb25af83502e1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:55:02.917142 18596 raft_consensus.cc:2258] T 0222627b936547e0951837de3e914df4 P 2b18011dc9b24cf587c3bb2237afc7e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:55:02.917626 18596 ts_tablet_manager.cc:1905] T 0222627b936547e0951837de3e914df4 P 2b18011dc9b24cf587c3bb2237afc7e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:55:02.918692 18593 ts_tablet_manager.cc:1918] T 0222627b936547e0951837de3e914df4 P bb2eb90f68da49f7b9fb25af83502e1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:55:02.918783 18593 log.cc:1196] T 0222627b936547e0951837de3e914df4 P bb2eb90f68da49f7b9fb25af83502e1f: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-1/wal/wals/0222627b936547e0951837de3e914df4
I20260929 10:55:02.919077 18593 ts_tablet_manager.cc:1939] T 0222627b936547e0951837de3e914df4 P bb2eb90f68da49f7b9fb25af83502e1f: Deleting consensus metadata
I20260929 10:55:02.919510 18596 ts_tablet_manager.cc:1918] T 0222627b936547e0951837de3e914df4 P 2b18011dc9b24cf587c3bb2237afc7e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:55:02.919615 18596 log.cc:1196] T 0222627b936547e0951837de3e914df4 P 2b18011dc9b24cf587c3bb2237afc7e6: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-0/wal/wals/0222627b936547e0951837de3e914df4
I20260929 10:55:02.919860 18596 ts_tablet_manager.cc:1939] T 0222627b936547e0951837de3e914df4 P 2b18011dc9b24cf587c3bb2237afc7e6: Deleting consensus metadata
W20260929 10:55:02.922597 18072 connection.cc:535] server connection from 127.0.0.1:54914 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260929 10:55:02.922799 18289 connection.cc:535] server connection from 127.0.0.1:46756 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260929 10:55:02.922994 18152 connection.cc:535] server connection from 127.0.0.1:33278 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260929 10:55:02.923179 18428 connection.cc:535] server connection from 127.0.0.1:41756 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 10:55:02.929360 18269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.913660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54928 (local address 127.10.238.252:37593)
0929 10:55:02.913701 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:02.913703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:02.921436 (+ 7733us) server_negotiation.cc:407] Connection header received
0929 10:55:02.921511 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:02.921512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:02.921523 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:02.921578 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:02.922131 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.922734 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.923740 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.923948 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.924030 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.925863 (+ 1833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:02.925865 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:02.925865 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:02.925873 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:02.927584 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:02.927964 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:02.927965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:02.927966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:02.928019 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:02.928201 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:02.928202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:02.928202 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:02.928328 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:02.928646 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:02.928887 (+ 241us) server_negotiation.cc:299] Negotiation successful
0929 10:55:02.929239 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 10:55:02.932412 18551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.924832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48020 (local address 127.10.238.253:45709)
0929 10:55:02.924875 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:02.924877 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:02.924991 (+ 114us) server_negotiation.cc:407] Connection header received
0929 10:55:02.925085 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:02.925086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:02.925097 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:02.925127 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:02.925865 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.926646 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.927719 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.928053 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.928094 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.929528 (+ 1434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:02.929529 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:02.929529 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:02.929536 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:02.931183 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:02.931590 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:02.931591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:02.931591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:02.931655 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:02.931850 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:02.931851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:02.931851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:02.931978 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:02.932108 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:02.932205 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 10:55:02.932290 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 10:55:02.933771 18553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:02.925125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54938 (local address 127.10.238.252:37593)
0929 10:55:02.925164 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:02.925166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:02.925353 (+ 187us) server_negotiation.cc:407] Connection header received
0929 10:55:02.925453 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:02.925454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:02.925468 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:02.925512 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:02.926471 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.927229 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.928315 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:02.928623 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:02.928665 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:02.930649 (+ 1984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:02.930650 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:02.930650 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:02.930657 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:02.932366 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:02.932965 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:02.932966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:02.932967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:02.933021 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:02.933246 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:02.933247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:02.933248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:02.933389 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:02.933496 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:02.933599 (+ 103us) server_negotiation.cc:299] Negotiation successful
0929 10:55:02.933676 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 10:55:03.910396 18603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:03.910128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39293 (local address 127.10.238.194:38445)
0929 10:55:03.910255 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:03.910331 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":70,"threads_started":1}
W20260929 10:55:03.910624 18403 heartbeater.cc:644] Failed to heartbeat to 127.10.238.254:39293 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.238.254:39293: connect: Connection refused (error 111)
I20260929 10:55:03.914422 18605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:03.914166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39293 (local address 127.10.238.195:37775)
0929 10:55:03.914289 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:03.914366 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":52,"threads_started":1}
I20260929 10:55:03.914551 18604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:03.914092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39293 (local address 127.10.238.193:53459)
0929 10:55:03.914427 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:03.914499 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":80,"threads_started":1}
W20260929 10:55:03.914798 18539 heartbeater.cc:644] Failed to heartbeat to 127.10.238.254:39293 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.238.254:39293: connect: Connection refused (error 111)
W20260929 10:55:03.914839 18266 heartbeater.cc:644] Failed to heartbeat to 127.10.238.254:39293 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.238.254:39293: connect: Connection refused (error 111)
I20260929 10:55:04.414445 18606 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82fdb61571ad46209f5fe7dca6fae4bf)
I20260929 10:55:04.414562 18606 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } } peers { permanent_uuid: "59092f98066b4395b785a2698bf678c7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45709 } attrs { promote: false } } peers { permanent_uuid: "952082091a5949e68504685e56cd3c94" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 37593 } attrs { promote: false } }
I20260929 10:55:04.414820 18606 leader_election.cc:288] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82fdb61571ad46209f5fe7dca6fae4bf (127.10.238.254:39293), 952082091a5949e68504685e56cd3c94 (127.10.238.252:37593)
I20260929 10:55:04.415174 18608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:04.414889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39293 (local address 127.0.0.1:55360)
0929 10:55:04.415074 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:04.415130 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":69,"threads_started":1}
W20260929 10:55:04.415315 17884 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39293: connect: Connection refused (error 111)
I20260929 10:55:04.415362 18610 raft_consensus.cc:482] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82fdb61571ad46209f5fe7dca6fae4bf)
I20260929 10:55:04.415459 18610 raft_consensus.cc:504] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } } peers { permanent_uuid: "59092f98066b4395b785a2698bf678c7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45709 } attrs { promote: false } } peers { permanent_uuid: "952082091a5949e68504685e56cd3c94" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 37593 } attrs { promote: false } }
I20260929 10:55:04.415673 18610 leader_election.cc:288] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82fdb61571ad46209f5fe7dca6fae4bf (127.10.238.254:39293), 59092f98066b4395b785a2698bf678c7 (127.10.238.253:45709)
I20260929 10:55:04.415753 18608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:04.415645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39293 (local address 127.0.0.1:55376)
0929 10:55:04.415695 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:04.415725 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 10:55:04.415941 18612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:04.415754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39293 (local address 127.0.0.1:55386)
0929 10:55:04.415848 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:04.415900 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":51,"threads_started":1}
W20260929 10:55:04.415999 17884 leader_election.cc:334] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 82fdb61571ad46209f5fe7dca6fae4bf (127.10.238.254:39293): Network error: Client connection negotiation failed: client connection to 127.10.238.254:39293: connect: Connection refused (error 111)
W20260929 10:55:04.416085 18071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39293: connect: Connection refused (error 111)
I20260929 10:55:04.416582 18612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:04.416478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39293 (local address 127.0.0.1:55390)
0929 10:55:04.416529 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:04.416554 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260929 10:55:04.416661 18071 leader_election.cc:334] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 82fdb61571ad46209f5fe7dca6fae4bf (127.10.238.254:39293): Network error: Client connection negotiation failed: client connection to 127.10.238.254:39293: connect: Connection refused (error 111)
I20260929 10:55:04.419339 18607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:04.414970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.252:37593 (local address 127.0.0.1:54940)
0929 10:55:04.415111 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:04.415122 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:04.415181 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:04.415433 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:04.415434 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:04.415437 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:55:04.415502 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:04.415504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:04.417237 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:04.417238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:04.418503 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:04.418506 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:04.419149 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:04.419150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:04.419198 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:04.419202 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 10:55:04.419247 (+ 45us) client_negotiation.cc:240] Negotiation successful
0929 10:55:04.419283 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":46,"threads_started":1}
I20260929 10:55:04.419373 18609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:04.415115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54940 (local address 127.10.238.252:37593)
0929 10:55:04.415228 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:04.415230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:04.415239 (+ 9us) server_negotiation.cc:407] Connection header received
0929 10:55:04.415284 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:04.415284 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:04.415298 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:04.415357 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:55:04.416319 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:04.417142 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:04.418588 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:04.419068 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:04.419142 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:04.419151 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:04.419263 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 10:55:04.419309 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":60,"threads_started":1}
I20260929 10:55:04.419592 18096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "59092f98066b4395b785a2698bf678c7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "952082091a5949e68504685e56cd3c94" is_pre_election: true
I20260929 10:55:04.419714 18096 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59092f98066b4395b785a2698bf678c7 in term 1.
I20260929 10:55:04.419719 18613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:04.415799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.253:45709 (local address 127.0.0.1:48034)
0929 10:55:04.415896 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:04.415911 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:04.415950 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:04.416605 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:04.416606 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:04.416609 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:55:04.416776 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:04.416780 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:04.417573 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:04.417574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:04.418733 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:04.418735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:04.419463 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:04.419464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:04.419526 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:04.419531 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 10:55:04.419563 (+ 32us) client_negotiation.cc:240] Negotiation successful
0929 10:55:04.419599 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":55,"threads_started":1}
I20260929 10:55:04.419839 18614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:04.415897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48034 (local address 127.10.238.253:45709)
0929 10:55:04.416444 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:04.416446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:04.416454 (+ 8us) server_negotiation.cc:407] Connection header received
0929 10:55:04.416472 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:04.416473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:04.416482 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:04.416579 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:55:04.416859 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:04.417501 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:04.418814 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:04.419361 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:04.419675 (+ 314us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:04.419684 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:04.419702 (+ 18us) server_negotiation.cc:299] Negotiation successful
0929 10:55:04.419748 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":70,"threads_started":1}
I20260929 10:55:04.419915 17885 leader_election.cc:302] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59092f98066b4395b785a2698bf678c7, 952082091a5949e68504685e56cd3c94; no voters: 82fdb61571ad46209f5fe7dca6fae4bf
I20260929 10:55:04.420053 18606 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 10:55:04.420094 18606 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 82fdb61571ad46209f5fe7dca6fae4bf)
I20260929 10:55:04.420122 18606 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 [term 1 FOLLOWER]: Advancing to term 2
I20260929 10:55:04.420816 18606 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } } peers { permanent_uuid: "59092f98066b4395b785a2698bf678c7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45709 } attrs { promote: false } } peers { permanent_uuid: "952082091a5949e68504685e56cd3c94" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 37593 } attrs { promote: false } }
I20260929 10:55:04.420966 18606 leader_election.cc:288] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 [CANDIDATE]: Term 2 election: Requested vote from peers 82fdb61571ad46209f5fe7dca6fae4bf (127.10.238.254:39293), 952082091a5949e68504685e56cd3c94 (127.10.238.252:37593)
I20260929 10:55:04.421046 17909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "952082091a5949e68504685e56cd3c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "59092f98066b4395b785a2698bf678c7" is_pre_election: true
I20260929 10:55:04.421101 18096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "59092f98066b4395b785a2698bf678c7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "952082091a5949e68504685e56cd3c94"
I20260929 10:55:04.421126 17909 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 952082091a5949e68504685e56cd3c94 in current term 2: Already voted for candidate 59092f98066b4395b785a2698bf678c7 in this term.
I20260929 10:55:04.421164 18096 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94 [term 1 FOLLOWER]: Advancing to term 2
I20260929 10:55:04.421168 18607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:04.421056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39293 (local address 127.0.0.1:55404)
0929 10:55:04.421087 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:04.421116 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260929 10:55:04.421310 18072 leader_election.cc:302] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 952082091a5949e68504685e56cd3c94; no voters: 59092f98066b4395b785a2698bf678c7, 82fdb61571ad46209f5fe7dca6fae4bf
I20260929 10:55:04.421497 18607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:04.421410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39293 (local address 127.0.0.1:55418)
0929 10:55:04.421440 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:04.421480 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260929 10:55:04.421546 17884 leader_election.cc:334] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 82fdb61571ad46209f5fe7dca6fae4bf (127.10.238.254:39293): Network error: Client connection negotiation failed: client connection to 127.10.238.254:39293: connect: Connection refused (error 111)
I20260929 10:55:04.421880 18096 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59092f98066b4395b785a2698bf678c7 in term 2.
I20260929 10:55:04.421975 18610 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 10:55:04.422075 17885 leader_election.cc:302] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59092f98066b4395b785a2698bf678c7, 952082091a5949e68504685e56cd3c94; no voters: 82fdb61571ad46209f5fe7dca6fae4bf
I20260929 10:55:04.422161 18606 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 10:55:04.422291 18606 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 [term 2 LEADER]: Becoming Leader. State: Replica: 59092f98066b4395b785a2698bf678c7, State: Running, Role: LEADER
I20260929 10:55:04.422348 18606 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } } peers { permanent_uuid: "59092f98066b4395b785a2698bf678c7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45709 } attrs { promote: false } } peers { permanent_uuid: "952082091a5949e68504685e56cd3c94" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 37593 } attrs { promote: false } }
I20260929 10:55:04.422715 18616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59092f98066b4395b785a2698bf678c7. Latest consensus state: current_term: 2 leader_uuid: "59092f98066b4395b785a2698bf678c7" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } } peers { permanent_uuid: "59092f98066b4395b785a2698bf678c7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45709 } attrs { promote: false } } peers { permanent_uuid: "952082091a5949e68504685e56cd3c94" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 37593 } attrs { promote: false } } }
I20260929 10:55:04.422818 18616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 [sys.catalog]: This master's current role is: LEADER
I20260929 10:55:04.422994 18618 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:55:04.423436 18618 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:55:04.423496 18618 catalog_manager.cc:1259] Loaded cluster ID: 8ee41e681375487bbdaa5339ed2563f0
I20260929 10:55:04.423513 18618 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:55:04.423687 18618 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:55:04.423771 18618 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7: Loaded TSK: 0
I20260929 10:55:04.423940 18618 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:55:04.471145 17899 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48020:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:55:04.472911 17899 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:55:04.474031 18607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:04.473912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39293 (local address 127.0.0.1:55438)
0929 10:55:04.473968 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:04.473999 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 10:55:04.474072 18096 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94 [term 2 FOLLOWER]: Refusing update from remote peer 59092f98066b4395b785a2698bf678c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260929 10:55:04.474371 18616 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "952082091a5949e68504685e56cd3c94" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 37593 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260929 10:55:04.474457 18607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:04.474334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39293 (local address 127.0.0.1:55448)
0929 10:55:04.474378 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:04.474408 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260929 10:55:04.474718 17884 consensus_peers.cc:487] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 -> Peer 82fdb61571ad46209f5fe7dca6fae4bf (127.10.238.254:39293): Couldn't send request to peer 82fdb61571ad46209f5fe7dca6fae4bf. Status: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39293: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 10:55:04.475056 18610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59092f98066b4395b785a2698bf678c7. Latest consensus state: current_term: 2 leader_uuid: "59092f98066b4395b785a2698bf678c7" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } } peers { permanent_uuid: "59092f98066b4395b785a2698bf678c7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45709 } attrs { promote: false } } peers { permanent_uuid: "952082091a5949e68504685e56cd3c94" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 37593 } attrs { promote: false } } }
I20260929 10:55:04.475163 18610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:55:04.475646 18606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "59092f98066b4395b785a2698bf678c7" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } } peers { permanent_uuid: "59092f98066b4395b785a2698bf678c7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45709 } attrs { promote: false } } peers { permanent_uuid: "952082091a5949e68504685e56cd3c94" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 37593 } attrs { promote: false } } }
I20260929 10:55:04.475744 18606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7 [sys.catalog]: This master's current role is: LEADER
I20260929 10:55:04.475814 18610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "59092f98066b4395b785a2698bf678c7" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "82fdb61571ad46209f5fe7dca6fae4bf" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39293 } } peers { permanent_uuid: "59092f98066b4395b785a2698bf678c7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45709 } attrs { promote: false } } peers { permanent_uuid: "952082091a5949e68504685e56cd3c94" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 37593 } attrs { promote: false } } }
I20260929 10:55:04.475891 18610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 952082091a5949e68504685e56cd3c94 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:55:04.475961 18621 mvcc.cc:205] Tried to move back new op lower bound from 7334622431123496960 to 7334622430914752512. Current Snapshot: MvccSnapshot[applied={T|T < 7334622431123496960 or (T in {7334622431123496960})}]
I20260929 10:55:04.481626 18608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:04.477587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:42961 (local address 127.0.0.1:33298)
0929 10:55:04.477701 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:04.477710 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:04.477755 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:04.478114 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:04.478115 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:04.478117 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:55:04.478171 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:04.478172 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:04.479193 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:04.479195 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:04.480452 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:04.480455 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:04.481364 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:04.481366 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:04.481439 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:04.481444 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 10:55:04.481477 (+ 33us) client_negotiation.cc:240] Negotiation successful
0929 10:55:04.481527 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260929 10:55:04.481946 18625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:04.477731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33298 (local address 127.10.238.193:42961)
0929 10:55:04.477976 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:04.477978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:04.477986 (+ 8us) server_negotiation.cc:407] Connection header received
0929 10:55:04.478015 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:04.478016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:04.478025 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:04.478091 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:55:04.478253 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:04.479085 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:04.480587 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:04.481229 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:04.481757 (+ 528us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:04.481769 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:04.481792 (+ 23us) server_negotiation.cc:299] Negotiation successful
0929 10:55:04.481854 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":62,"threads_started":1}
I20260929 10:55:04.482182 18607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:04.477493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.195:45979 (local address 127.0.0.1:41760)
0929 10:55:04.477612 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:04.477623 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:04.477689 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:04.478660 (+ 971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:04.478660 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:04.478661 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:55:04.478693 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:04.478694 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:04.479786 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:04.479787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:04.481058 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:04.481061 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:04.481934 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:04.481935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:04.482002 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:04.482009 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 10:55:04.482055 (+ 46us) client_negotiation.cc:240] Negotiation successful
0929 10:55:04.482099 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260929 10:55:04.482544 18199 tablet_service.cc:1467] Processing CreateTablet for tablet 4c3a634d34a247728a3c6b16c5ada73a (DEFAULT_TABLE table=test-table [id=2b61ab573fe94af89e1bad0a0b4546b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:55:04.482620 18626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:04.477796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41760 (local address 127.10.238.195:45979)
0929 10:55:04.478518 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:04.478520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:04.478528 (+ 8us) server_negotiation.cc:407] Connection header received
0929 10:55:04.478557 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:04.478558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:04.478570 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:04.478818 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:55:04.478861 (+ 43us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:04.479679 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:04.481192 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:04.481836 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:04.482435 (+ 599us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:04.482448 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:04.482469 (+ 21us) server_negotiation.cc:299] Negotiation successful
0929 10:55:04.482528 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":60,"threads_started":1}
I20260929 10:55:04.482712 18199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c3a634d34a247728a3c6b16c5ada73a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:55:04.483879 18629 tablet_bootstrap.cc:492] T 4c3a634d34a247728a3c6b16c5ada73a P 2b18011dc9b24cf587c3bb2237afc7e6: Bootstrap starting.
I20260929 10:55:04.484305 18629 tablet_bootstrap.cc:654] T 4c3a634d34a247728a3c6b16c5ada73a P 2b18011dc9b24cf587c3bb2237afc7e6: Neither blocks nor log segments found. Creating new log.
I20260929 10:55:04.485064 18629 tablet_bootstrap.cc:492] T 4c3a634d34a247728a3c6b16c5ada73a P 2b18011dc9b24cf587c3bb2237afc7e6: No bootstrap required, opened a new log
I20260929 10:55:04.485128 18629 ts_tablet_manager.cc:1397] T 4c3a634d34a247728a3c6b16c5ada73a P 2b18011dc9b24cf587c3bb2237afc7e6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 10:55:04.485181 18627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:04.477591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.194:40983 (local address 127.0.0.1:46766)
0929 10:55:04.478299 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:04.478341 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:04.478399 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:04.478595 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:04.478595 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:04.478597 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:55:04.478660 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:04.478661 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:04.479770 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:04.479771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:04.481365 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:04.481367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:04.484881 (+ 3514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:04.484882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:04.484953 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:04.484957 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 10:55:04.485043 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 10:55:04.485097 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"mutex_wait_us":581,"thread_start_us":76,"threads_started":1}
I20260929 10:55:04.485275 18629 raft_consensus.cc:348] T 4c3a634d34a247728a3c6b16c5ada73a P 2b18011dc9b24cf587c3bb2237afc7e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c69ddb2aee461092fbe82090413694" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 45979 } } peers { permanent_uuid: "bb2eb90f68da49f7b9fb25af83502e1f" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40983 } } peers { permanent_uuid: "2b18011dc9b24cf587c3bb2237afc7e6" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 42961 } }
I20260929 10:55:04.485369 18629 raft_consensus.cc:374] T 4c3a634d34a247728a3c6b16c5ada73a P 2b18011dc9b24cf587c3bb2237afc7e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:55:04.485363 18472 tablet_service.cc:1467] Processing CreateTablet for tablet 4c3a634d34a247728a3c6b16c5ada73a (DEFAULT_TABLE table=test-table [id=2b61ab573fe94af89e1bad0a0b4546b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:55:04.485399 18629 raft_consensus.cc:729] T 4c3a634d34a247728a3c6b16c5ada73a P 2b18011dc9b24cf587c3bb2237afc7e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b18011dc9b24cf587c3bb2237afc7e6, State: Initialized, Role: FOLLOWER
I20260929 10:55:04.485486 18472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c3a634d34a247728a3c6b16c5ada73a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:55:04.485466 18629 consensus_queue.cc:260] T 4c3a634d34a247728a3c6b16c5ada73a P 2b18011dc9b24cf587c3bb2237afc7e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c69ddb2aee461092fbe82090413694" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 45979 } } peers { permanent_uuid: "bb2eb90f68da49f7b9fb25af83502e1f" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40983 } } peers { permanent_uuid: "2b18011dc9b24cf587c3bb2237afc7e6" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 42961 } }
I20260929 10:55:04.485570 18628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:04.478050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46766 (local address 127.10.238.194:40983)
0929 10:55:04.478342 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:04.478345 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:04.478384 (+ 39us) server_negotiation.cc:407] Connection header received
0929 10:55:04.478467 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:04.478468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:04.478480 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:04.478544 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:55:04.478744 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:04.479666 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:04.481458 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:04.484748 (+ 3290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:04.484830 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:04.484840 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:04.485053 (+ 213us) server_negotiation.cc:299] Negotiation successful
0929 10:55:04.485100 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":55,"threads_started":1}
I20260929 10:55:04.485613 18629 ts_tablet_manager.cc:1428] T 4c3a634d34a247728a3c6b16c5ada73a P 2b18011dc9b24cf587c3bb2237afc7e6: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260929 10:55:04.485968 18336 tablet_service.cc:1467] Processing CreateTablet for tablet 4c3a634d34a247728a3c6b16c5ada73a (DEFAULT_TABLE table=test-table [id=2b61ab573fe94af89e1bad0a0b4546b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:55:04.486099 18336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c3a634d34a247728a3c6b16c5ada73a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:55:04.486606 18632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:04.485879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39293 (local address 127.10.238.193:33409)
0929 10:55:04.486112 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:04.486199 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":111,"threads_started":1}
I20260929 10:55:04.487959 18633 tablet_bootstrap.cc:492] T 4c3a634d34a247728a3c6b16c5ada73a P 99c69ddb2aee461092fbe82090413694: Bootstrap starting.
I20260929 10:55:04.487985 18632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:04.487860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39293 (local address 127.10.238.193:55933)
0929 10:55:04.487926 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:04.487956 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 10:55:04.488003 18634 tablet_bootstrap.cc:492] T 4c3a634d34a247728a3c6b16c5ada73a P bb2eb90f68da49f7b9fb25af83502e1f: Bootstrap starting.
I20260929 10:55:04.488381 18633 tablet_bootstrap.cc:654] T 4c3a634d34a247728a3c6b16c5ada73a P 99c69ddb2aee461092fbe82090413694: Neither blocks nor log segments found. Creating new log.
I20260929 10:55:04.488499 18634 tablet_bootstrap.cc:654] T 4c3a634d34a247728a3c6b16c5ada73a P bb2eb90f68da49f7b9fb25af83502e1f: Neither blocks nor log segments found. Creating new log.
I20260929 10:55:04.489027 18633 tablet_bootstrap.cc:492] T 4c3a634d34a247728a3c6b16c5ada73a P 99c69ddb2aee461092fbe82090413694: No bootstrap required, opened a new log
I20260929 10:55:04.489089 18633 ts_tablet_manager.cc:1397] T 4c3a634d34a247728a3c6b16c5ada73a P 99c69ddb2aee461092fbe82090413694: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 10:55:04.489279 18633 raft_consensus.cc:348] T 4c3a634d34a247728a3c6b16c5ada73a P 99c69ddb2aee461092fbe82090413694 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c69ddb2aee461092fbe82090413694" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 45979 } } peers { permanent_uuid: "bb2eb90f68da49f7b9fb25af83502e1f" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40983 } } peers { permanent_uuid: "2b18011dc9b24cf587c3bb2237afc7e6" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 42961 } }
I20260929 10:55:04.489358 18633 raft_consensus.cc:374] T 4c3a634d34a247728a3c6b16c5ada73a P 99c69ddb2aee461092fbe82090413694 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:55:04.489382 18633 raft_consensus.cc:729] T 4c3a634d34a247728a3c6b16c5ada73a P 99c69ddb2aee461092fbe82090413694 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99c69ddb2aee461092fbe82090413694, State: Initialized, Role: FOLLOWER
I20260929 10:55:04.489614 18634 tablet_bootstrap.cc:492] T 4c3a634d34a247728a3c6b16c5ada73a P bb2eb90f68da49f7b9fb25af83502e1f: No bootstrap required, opened a new log
I20260929 10:55:04.489670 18634 ts_tablet_manager.cc:1397] T 4c3a634d34a247728a3c6b16c5ada73a P bb2eb90f68da49f7b9fb25af83502e1f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 10:55:04.489799 18634 raft_consensus.cc:348] T 4c3a634d34a247728a3c6b16c5ada73a P bb2eb90f68da49f7b9fb25af83502e1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c69ddb2aee461092fbe82090413694" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 45979 } } peers { permanent_uuid: "bb2eb90f68da49f7b9fb25af83502e1f" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40983 } } peers { permanent_uuid: "2b18011dc9b24cf587c3bb2237afc7e6" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 42961 } }
I20260929 10:55:04.489853 18634 raft_consensus.cc:374] T 4c3a634d34a247728a3c6b16c5ada73a P bb2eb90f68da49f7b9fb25af83502e1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:55:04.489868 18634 raft_consensus.cc:729] T 4c3a634d34a247728a3c6b16c5ada73a P bb2eb90f68da49f7b9fb25af83502e1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb2eb90f68da49f7b9fb25af83502e1f, State: Initialized, Role: FOLLOWER
I20260929 10:55:04.489931 18634 consensus_queue.cc:260] T 4c3a634d34a247728a3c6b16c5ada73a P bb2eb90f68da49f7b9fb25af83502e1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c69ddb2aee461092fbe82090413694" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 45979 } } peers { permanent_uuid: "bb2eb90f68da49f7b9fb25af83502e1f" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40983 } } peers { permanent_uuid: "2b18011dc9b24cf587c3bb2237afc7e6" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 42961 } }
I20260929 10:55:04.490105 18634 ts_tablet_manager.cc:1428] T 4c3a634d34a247728a3c6b16c5ada73a P bb2eb90f68da49f7b9fb25af83502e1f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 10:55:04.490484 18638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:04.490273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39293 (local address 127.10.238.194:45689)
0929 10:55:04.490401 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:04.490436 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":86,"threads_started":1}
I20260929 10:55:04.490650 18404 heartbeater.cc:503] Master 127.10.238.253:45709 was elected leader, sending a full tablet report...
I20260929 10:55:04.490399 18633 consensus_queue.cc:260] T 4c3a634d34a247728a3c6b16c5ada73a P 99c69ddb2aee461092fbe82090413694 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c69ddb2aee461092fbe82090413694" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 45979 } } peers { permanent_uuid: "bb2eb90f68da49f7b9fb25af83502e1f" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40983 } } peers { permanent_uuid: "2b18011dc9b24cf587c3bb2237afc7e6" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 42961 } }
I20260929 10:55:04.490993 18638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:04.490868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39293 (local address 127.10.238.194:59133)
0929 10:55:04.490905 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:04.490928 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260929 10:55:04.491149 18633 ts_tablet_manager.cc:1428] T 4c3a634d34a247728a3c6b16c5ada73a P 99c69ddb2aee461092fbe82090413694: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 10:55:04.491541 18640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:04.491306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39293 (local address 127.10.238.195:50941)
0929 10:55:04.491425 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:04.491468 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":77,"threads_started":1}
I20260929 10:55:04.491657 18540 heartbeater.cc:503] Master 127.10.238.253:45709 was elected leader, sending a full tablet report...
I20260929 10:55:04.491842 18640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:04.491753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39293 (local address 127.10.238.195:55877)
0929 10:55:04.491791 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:04.491822 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260929 10:55:04.706077 18637 raft_consensus.cc:482] T 4c3a634d34a247728a3c6b16c5ada73a P bb2eb90f68da49f7b9fb25af83502e1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:55:04.706194 18637 raft_consensus.cc:504] T 4c3a634d34a247728a3c6b16c5ada73a P bb2eb90f68da49f7b9fb25af83502e1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c69ddb2aee461092fbe82090413694" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 45979 } } peers { permanent_uuid: "bb2eb90f68da49f7b9fb25af83502e1f" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40983 } } peers { permanent_uuid: "2b18011dc9b24cf587c3bb2237afc7e6" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 42961 } }
I20260929 10:55:04.706555 18637 leader_election.cc:288] T 4c3a634d34a247728a3c6b16c5ada73a P bb2eb90f68da49f7b9fb25af83502e1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99c69ddb2aee461092fbe82090413694 (127.10.238.195:45979), 2b18011dc9b24cf587c3bb2237afc7e6 (127.10.238.193:42961)
I20260929 10:55:04.709621 18626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:04.706867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:55785 (local address 127.10.238.195:45979)
0929 10:55:04.706912 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:04.706915 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:04.707050 (+ 135us) server_negotiation.cc:407] Connection header received
0929 10:55:04.707120 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:04.707121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:04.707131 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:04.707202 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:55:04.707360 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:04.707839 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:04.708813 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:04.709194 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:04.709361 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:04.709373 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:04.709510 (+ 137us) server_negotiation.cc:299] Negotiation successful
0929 10:55:04.709564 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 10:55:04.709626 18641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:04.706761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.195:45979 (local address 127.10.238.194:55785)
0929 10:55:04.706953 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:04.706963 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:04.707062 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:04.707206 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:04.707207 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:04.707209 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:55:04.707280 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:04.707282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:04.707922 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:04.707923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:04.708709 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:04.708711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:04.709307 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:04.709308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:04.709412 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:04.709416 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 10:55:04.709489 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 10:55:04.709531 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":111,"threads_started":1}
I20260929 10:55:04.710038 18492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c3a634d34a247728a3c6b16c5ada73a" candidate_uuid: "bb2eb90f68da49f7b9fb25af83502e1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99c69ddb2aee461092fbe82090413694" is_pre_election: true
I20260929 10:55:04.710143 18492 raft_consensus.cc:2454] T 4c3a634d34a247728a3c6b16c5ada73a P 99c69ddb2aee461092fbe82090413694 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb2eb90f68da49f7b9fb25af83502e1f in term 0.
I20260929 10:55:04.710156 18638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:04.706733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:42961 (local address 127.10.238.194:53727)
0929 10:55:04.706845 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:04.706860 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:04.706934 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:04.707181 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:04.707182 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:04.707185 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:55:04.707280 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:04.707282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:04.708027 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:04.708028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:04.708822 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:04.708824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:04.709595 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:04.709597 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:04.709942 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:04.709946 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 10:55:04.710036 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 10:55:04.710082 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260929 10:55:04.710165 18625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:04.706812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:53727 (local address 127.10.238.193:42961)
0929 10:55:04.706871 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:04.706873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:04.706931 (+ 58us) server_negotiation.cc:407] Connection header received
0929 10:55:04.707042 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:04.707045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:04.707072 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:04.707172 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:55:04.707365 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:04.707923 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:04.708961 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:04.709466 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:04.709560 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:04.709570 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:04.710047 (+ 477us) server_negotiation.cc:299] Negotiation successful
0929 10:55:04.710091 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 10:55:04.710325 18289 leader_election.cc:302] T 4c3a634d34a247728a3c6b16c5ada73a P bb2eb90f68da49f7b9fb25af83502e1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99c69ddb2aee461092fbe82090413694, bb2eb90f68da49f7b9fb25af83502e1f; no voters:
I20260929 10:55:04.710455 18637 raft_consensus.cc:2781] T 4c3a634d34a247728a3c6b16c5ada73a P bb2eb90f68da49f7b9fb25af83502e1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:55:04.710498 18637 raft_consensus.cc:482] T 4c3a634d34a247728a3c6b16c5ada73a P bb2eb90f68da49f7b9fb25af83502e1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:55:04.710527 18637 raft_consensus.cc:3037] T 4c3a634d34a247728a3c6b16c5ada73a P bb2eb90f68da49f7b9fb25af83502e1f [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:55:04.710462 18219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c3a634d34a247728a3c6b16c5ada73a" candidate_uuid: "bb2eb90f68da49f7b9fb25af83502e1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b18011dc9b24cf587c3bb2237afc7e6" is_pre_election: true
I20260929 10:55:04.710595 18219 raft_consensus.cc:2454] T 4c3a634d34a247728a3c6b16c5ada73a P 2b18011dc9b24cf587c3bb2237afc7e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb2eb90f68da49f7b9fb25af83502e1f in term 0.
I20260929 10:55:04.711424 18637 raft_consensus.cc:504] T 4c3a634d34a247728a3c6b16c5ada73a P bb2eb90f68da49f7b9fb25af83502e1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c69ddb2aee461092fbe82090413694" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 45979 } } peers { permanent_uuid: "bb2eb90f68da49f7b9fb25af83502e1f" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40983 } } peers { permanent_uuid: "2b18011dc9b24cf587c3bb2237afc7e6" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 42961 } }
I20260929 10:55:04.711583 18637 leader_election.cc:288] T 4c3a634d34a247728a3c6b16c5ada73a P bb2eb90f68da49f7b9fb25af83502e1f [CANDIDATE]: Term 1 election: Requested vote from peers 99c69ddb2aee461092fbe82090413694 (127.10.238.195:45979), 2b18011dc9b24cf587c3bb2237afc7e6 (127.10.238.193:42961)
I20260929 10:55:04.711714 18492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c3a634d34a247728a3c6b16c5ada73a" candidate_uuid: "bb2eb90f68da49f7b9fb25af83502e1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99c69ddb2aee461092fbe82090413694"
I20260929 10:55:04.711736 18219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c3a634d34a247728a3c6b16c5ada73a" candidate_uuid: "bb2eb90f68da49f7b9fb25af83502e1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b18011dc9b24cf587c3bb2237afc7e6"
I20260929 10:55:04.711786 18492 raft_consensus.cc:3037] T 4c3a634d34a247728a3c6b16c5ada73a P 99c69ddb2aee461092fbe82090413694 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:55:04.711796 18219 raft_consensus.cc:3037] T 4c3a634d34a247728a3c6b16c5ada73a P 2b18011dc9b24cf587c3bb2237afc7e6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:55:04.712553 18219 raft_consensus.cc:2454] T 4c3a634d34a247728a3c6b16c5ada73a P 2b18011dc9b24cf587c3bb2237afc7e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb2eb90f68da49f7b9fb25af83502e1f in term 1.
I20260929 10:55:04.712553 18492 raft_consensus.cc:2454] T 4c3a634d34a247728a3c6b16c5ada73a P 99c69ddb2aee461092fbe82090413694 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb2eb90f68da49f7b9fb25af83502e1f in term 1.
I20260929 10:55:04.712751 18289 leader_election.cc:302] T 4c3a634d34a247728a3c6b16c5ada73a P bb2eb90f68da49f7b9fb25af83502e1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99c69ddb2aee461092fbe82090413694, bb2eb90f68da49f7b9fb25af83502e1f; no voters:
I20260929 10:55:04.712862 18637 raft_consensus.cc:2781] T 4c3a634d34a247728a3c6b16c5ada73a P bb2eb90f68da49f7b9fb25af83502e1f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:55:04.712996 18637 raft_consensus.cc:686] T 4c3a634d34a247728a3c6b16c5ada73a P bb2eb90f68da49f7b9fb25af83502e1f [term 1 LEADER]: Becoming Leader. State: Replica: bb2eb90f68da49f7b9fb25af83502e1f, State: Running, Role: LEADER
I20260929 10:55:04.713049 18637 consensus_queue.cc:237] T 4c3a634d34a247728a3c6b16c5ada73a P bb2eb90f68da49f7b9fb25af83502e1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c69ddb2aee461092fbe82090413694" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 45979 } } peers { permanent_uuid: "bb2eb90f68da49f7b9fb25af83502e1f" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40983 } } peers { permanent_uuid: "2b18011dc9b24cf587c3bb2237afc7e6" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 42961 } }
I20260929 10:55:04.713497 18638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:04.713395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39293 (local address 127.10.238.194:58033)
0929 10:55:04.713446 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:04.713477 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260929 10:55:04.714224 17899 catalog_manager.cc:5495] T 4c3a634d34a247728a3c6b16c5ada73a P bb2eb90f68da49f7b9fb25af83502e1f reported cstate change: term changed from 0 to 1, leader changed from <none> to bb2eb90f68da49f7b9fb25af83502e1f (127.10.238.194). New cstate: current_term: 1 leader_uuid: "bb2eb90f68da49f7b9fb25af83502e1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c69ddb2aee461092fbe82090413694" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb2eb90f68da49f7b9fb25af83502e1f" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b18011dc9b24cf587c3bb2237afc7e6" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 42961 } health_report { overall_health: UNKNOWN } } }
I20260929 10:55:04.765532 18628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:04.763202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46782 (local address 127.10.238.194:40983)
0929 10:55:04.763265 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:04.763267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:04.763276 (+ 9us) server_negotiation.cc:407] Connection header received
0929 10:55:04.763324 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:04.763325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:04.763343 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:04.763405 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:55:04.763529 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:04.764002 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:04.764716 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:04.764913 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:04.764975 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:04.765104 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:55:04.765207 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:55:04.765357 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:04.765389 (+ 32us) server_negotiation.cc:299] Negotiation successful
0929 10:55:04.765412 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 10:55:04.766847 18492 raft_consensus.cc:1261] T 4c3a634d34a247728a3c6b16c5ada73a P 99c69ddb2aee461092fbe82090413694 [term 1 FOLLOWER]: Refusing update from remote peer bb2eb90f68da49f7b9fb25af83502e1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:55:04.766870 18219 raft_consensus.cc:1261] T 4c3a634d34a247728a3c6b16c5ada73a P 2b18011dc9b24cf587c3bb2237afc7e6 [term 1 FOLLOWER]: Refusing update from remote peer bb2eb90f68da49f7b9fb25af83502e1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:55:04.767122 18637 consensus_queue.cc:1035] T 4c3a634d34a247728a3c6b16c5ada73a P bb2eb90f68da49f7b9fb25af83502e1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "99c69ddb2aee461092fbe82090413694" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:55:04.767266 18642 consensus_queue.cc:1035] T 4c3a634d34a247728a3c6b16c5ada73a P bb2eb90f68da49f7b9fb25af83502e1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b18011dc9b24cf587c3bb2237afc7e6" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 42961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:55:04.768020 18267 heartbeater.cc:503] Master 127.10.238.253:45709 was elected leader, sending a full tablet report...
I20260929 10:55:04.768224 18640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:04.768118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39293 (local address 127.10.238.195:58455)
0929 10:55:04.768156 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:04.768189 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260929 10:55:04.768393 18632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:04.768228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39293 (local address 127.10.238.193:55121)
0929 10:55:04.768335 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:04.768363 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260929 10:55:04.770390 18632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:04.770105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39293 (local address 127.10.238.193:49631)
0929 10:55:04.770330 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:04.770362 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209}
I20260929 10:55:04.770802 18640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:04.770693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39293 (local address 127.10.238.195:51089)
0929 10:55:04.770747 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:04.770771 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260929 10:55:04.772462 17898 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:55:04.772555 17898 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:55:04.772687 18638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:04.769959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39293 (local address 127.10.238.194:42131)
0929 10:55:04.772603 (+ 2644us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:04.772661 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2629}
I20260929 10:55:04.773898 17898 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 59092f98066b4395b785a2698bf678c7: Sending DeleteTablet for 3 replicas of tablet 4c3a634d34a247728a3c6b16c5ada73a
I20260929 10:55:04.774276 18472 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c3a634d34a247728a3c6b16c5ada73a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:55:04 UTC) from {username='oryx', principal='oryx/127.10.238.253@KRBTEST.COM'} at 127.0.0.1:41760
I20260929 10:55:04.774348 18336 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c3a634d34a247728a3c6b16c5ada73a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:55:04 UTC) from {username='oryx', principal='oryx/127.10.238.253@KRBTEST.COM'} at 127.0.0.1:46766
I20260929 10:55:04.774463 18655 tablet_replica.cc:330] T 4c3a634d34a247728a3c6b16c5ada73a P 99c69ddb2aee461092fbe82090413694: stopping tablet replica
I20260929 10:55:04.774525 18655 raft_consensus.cc:2229] T 4c3a634d34a247728a3c6b16c5ada73a P 99c69ddb2aee461092fbe82090413694 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:55:04.774577 18655 raft_consensus.cc:2258] T 4c3a634d34a247728a3c6b16c5ada73a P 99c69ddb2aee461092fbe82090413694 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:55:04.774695 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 18046
I20260929 10:55:04.775676 18655 ts_tablet_manager.cc:1905] T 4c3a634d34a247728a3c6b16c5ada73a P 99c69ddb2aee461092fbe82090413694: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:55:04.777377 18655 ts_tablet_manager.cc:1918] T 4c3a634d34a247728a3c6b16c5ada73a P 99c69ddb2aee461092fbe82090413694: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:55:04.777448 18655 log.cc:1196] T 4c3a634d34a247728a3c6b16c5ada73a P 99c69ddb2aee461092fbe82090413694: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790679274510869-11195-0/minicluster-data/ts-2/wal/wals/4c3a634d34a247728a3c6b16c5ada73a
I20260929 10:55:04.777690 18655 ts_tablet_manager.cc:1939] T 4c3a634d34a247728a3c6b16c5ada73a P 99c69ddb2aee461092fbe82090413694: Deleting consensus metadata
I20260929 10:55:04.778013 17883 catalog_manager.cc:4841] TS 99c69ddb2aee461092fbe82090413694 (127.10.238.195:45979): tablet 4c3a634d34a247728a3c6b16c5ada73a (table test-table [id=2b61ab573fe94af89e1bad0a0b4546b2]) successfully deleted
W20260929 10:55:04.784660 17885 connection.cc:535] client connection to 127.10.238.193:42961 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 10:55:04.784811 18656 tablet_replica.cc:330] T 4c3a634d34a247728a3c6b16c5ada73a P bb2eb90f68da49f7b9fb25af83502e1f: stopping tablet replica
I20260929 10:55:04.784876 18656 raft_consensus.cc:2229] T 4c3a634d34a247728a3c6b16c5ada73a P bb2eb90f68da49f7b9fb25af83502e1f [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:55:04.784977 18656 raft_consensus.cc:2258] T 4c3a634d34a247728a3c6b16c5ada73a P bb2eb90f68da49f7b9fb25af83502e1f [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 10:55:04.785208 18627 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:04.784975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:42961 (local address 127.0.0.1:33300)
0929 10:55:04.785020 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:04.785032 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:04.785080 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:04.785158 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.193:42961: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":25}
W20260929 10:55:04.785367 17885 catalog_manager.cc:4584] TS 2b18011dc9b24cf587c3bb2237afc7e6 (127.10.238.193:42961): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4c3a634d34a247728a3c6b16c5ada73a: Network error: Client connection negotiation failed: client connection to 127.10.238.193:42961: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260929 10:55:04.785456 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 18273
I20260929 10:55:04.790158 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 18409
I20260929 10:55:04.790242 18627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:04.790068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.194:40983 (local address 127.0.0.1:46794)
0929 10:55:04.790146 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:04.790174 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.194:40983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260929 10:55:04.797519 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 17765
I20260929 10:55:04.803874 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 17953
2026-09-29T10:55:04Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4992 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:55:04 dist-test-slave-8nrt krb5kdc[18659](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:55:04 dist-test-slave-8nrt krb5kdc[18659](info): set up 2 sockets
Sep 29 10:55:04 dist-test-slave-8nrt krb5kdc[18659](info): commencing operation
krb5kdc: starting...
W20260929 10:55:05.853762 11195 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Sep 29 10:55:05 dist-test-slave-8nrt krb5kdc[18659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:55:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:55:05Z Disabled control of system clock
WARNING: no policy specified for oryx/127.10.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.254.keytab.
Entry for principal oryx/127.10.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.254.keytab.
I20260929 10:55:05.976241 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:36571
--webserver_interface=127.10.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.254.keytab
--principal=oryx/127.10.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:41141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:36571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:55:06.057428 18672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:55:06.057639 18672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:55:06.057662 18672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:55:06.058871 18672 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:55:06.058918 18672 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 10:55:06.058933 18672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:55:06.058945 18672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:55:06.058959 18672 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:55:06.058970 18672 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:55:06.060575 18672 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:41141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:36571
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.254.keytab
--principal=oryx/127.10.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:36571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:55:06.060868 18672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:55:06.061126 18672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:55:06.066113 18677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:06.066092 18680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:06.066143 18678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:55:06.066432 18672 server_base.cc:1034] running on GCE node
I20260929 10:55:06.066701 18672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:55:06.067000 18672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:55:06.068202 18672 hybrid_clock.cc:648] HybridClock initialized: now 1790679306068202 us; error 47 us; skew 500 ppm
Sep 29 10:55:06 dist-test-slave-8nrt krb5kdc[18659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679306, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:55:06.071149 18672 init.cc:377] Logged in from keytab as oryx/127.10.238.254@KRBTEST.COM (short username oryx)
I20260929 10:55:06.071641 18672 webserver.cc:460] Webserver started at http://127.10.238.254:36317/ using document root <none> and password file <none>
I20260929 10:55:06.071893 18672 fs_manager.cc:359] Metadata directory not provided
I20260929 10:55:06.071941 18672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:55:06.072072 18672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:55:06.073246 18672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/master-0/data/instance:
uuid: "f5072e103f2c4e39a27e638aa276acb7"
format_stamp: "Formatted at 2026-09-29 10:55:06 on dist-test-slave-8nrt"
server_key: "e638fd9eae4113b2d1d6e0f4c18b790c"
server_key_iv: "58f6705428685e716e821913b9cdc0eb"
server_key_version: "encryptionkey@0"
I20260929 10:55:06.073706 18672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/master-0/wal/instance:
uuid: "f5072e103f2c4e39a27e638aa276acb7"
format_stamp: "Formatted at 2026-09-29 10:55:06 on dist-test-slave-8nrt"
server_key: "e638fd9eae4113b2d1d6e0f4c18b790c"
server_key_iv: "58f6705428685e716e821913b9cdc0eb"
server_key_version: "encryptionkey@0"
I20260929 10:55:06.075156 18672 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 10:55:06.076085 18687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:06.076372 18672 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:55:06.076574 18672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "f5072e103f2c4e39a27e638aa276acb7"
format_stamp: "Formatted at 2026-09-29 10:55:06 on dist-test-slave-8nrt"
server_key: "e638fd9eae4113b2d1d6e0f4c18b790c"
server_key_iv: "58f6705428685e716e821913b9cdc0eb"
server_key_version: "encryptionkey@0"
I20260929 10:55:06.076690 18672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:55:06.091912 18672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:55:06.094460 18672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:55:06.094630 18672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:55:06.099305 18672 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.254:36571
I20260929 10:55:06.099402 18749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.254:36571 every 8 connection(s)
I20260929 10:55:06.099721 18672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:55:06.100435 18750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:55:06.101212 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 18672
I20260929 10:55:06.101363 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/master-0/wal/instance
I20260929 10:55:06.101567 11195 external_mini_cluster.cc:1428] Setting key cc12d7b4846b3998fbfccadeeba15326
I20260929 10:55:06.103751 18750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5072e103f2c4e39a27e638aa276acb7: Bootstrap starting.
I20260929 10:55:06.104372 18750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5072e103f2c4e39a27e638aa276acb7: Neither blocks nor log segments found. Creating new log.
I20260929 10:55:06.104660 18750 log.cc:824] T 00000000000000000000000000000000 P f5072e103f2c4e39a27e638aa276acb7: Log is configured to *not* fsync() on all Append() calls
I20260929 10:55:06.105448 18750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5072e103f2c4e39a27e638aa276acb7: No bootstrap required, opened a new log
I20260929 10:55:06.108029 18750 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5072e103f2c4e39a27e638aa276acb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5072e103f2c4e39a27e638aa276acb7" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 36571 } }
I20260929 10:55:06.108162 18750 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5072e103f2c4e39a27e638aa276acb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:55:06.108186 18750 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5072e103f2c4e39a27e638aa276acb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5072e103f2c4e39a27e638aa276acb7, State: Initialized, Role: FOLLOWER
I20260929 10:55:06.108321 18750 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5072e103f2c4e39a27e638aa276acb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5072e103f2c4e39a27e638aa276acb7" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 36571 } }
Sep 29 10:55:06 dist-test-slave-8nrt krb5kdc[18659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM
I20260929 10:55:06.108395 18750 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5072e103f2c4e39a27e638aa276acb7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:55:06.108443 18750 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5072e103f2c4e39a27e638aa276acb7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:55:06.108493 18750 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5072e103f2c4e39a27e638aa276acb7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:55:06.109031 18750 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5072e103f2c4e39a27e638aa276acb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5072e103f2c4e39a27e638aa276acb7" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 36571 } }
I20260929 10:55:06.109151 18750 leader_election.cc:302] T 00000000000000000000000000000000 P f5072e103f2c4e39a27e638aa276acb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5072e103f2c4e39a27e638aa276acb7; no voters:
I20260929 10:55:06.109334 18750 leader_election.cc:288] T 00000000000000000000000000000000 P f5072e103f2c4e39a27e638aa276acb7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:55:06.109477 18755 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5072e103f2c4e39a27e638aa276acb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:55:06.109607 18750 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5072e103f2c4e39a27e638aa276acb7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:55:06.109632 18755 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5072e103f2c4e39a27e638aa276acb7 [term 1 LEADER]: Becoming Leader. State: Replica: f5072e103f2c4e39a27e638aa276acb7, State: Running, Role: LEADER
I20260929 10:55:06.109753 18755 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5072e103f2c4e39a27e638aa276acb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5072e103f2c4e39a27e638aa276acb7" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 36571 } }
I20260929 10:55:06.110436 18755 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5072e103f2c4e39a27e638aa276acb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5072e103f2c4e39a27e638aa276acb7. Latest consensus state: current_term: 1 leader_uuid: "f5072e103f2c4e39a27e638aa276acb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5072e103f2c4e39a27e638aa276acb7" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 36571 } } }
I20260929 10:55:06.110518 18755 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5072e103f2c4e39a27e638aa276acb7 [sys.catalog]: This master's current role is: LEADER
I20260929 10:55:06.110716 18756 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5072e103f2c4e39a27e638aa276acb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5072e103f2c4e39a27e638aa276acb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5072e103f2c4e39a27e638aa276acb7" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 36571 } } }
I20260929 10:55:06.110814 18756 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5072e103f2c4e39a27e638aa276acb7 [sys.catalog]: This master's current role is: LEADER
W20260929 10:55:06.111472 18770 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f5072e103f2c4e39a27e638aa276acb7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:55:06.111770 18770 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:55:06.112089 18765 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:55:06.112315 18765 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:55:06.113090 18753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:06.102104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33020 (local address 127.10.238.254:36571)
0929 10:55:06.102282 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:06.102290 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:06.102315 (+ 25us) server_negotiation.cc:407] Connection header received
0929 10:55:06.102760 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:06.102766 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:06.102812 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:06.102921 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:06.103745 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.104718 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.105764 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.106152 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.106233 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:06.108895 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:06.108900 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:06.108901 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:06.108913 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:06.111315 (+ 2402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:06.111757 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:06.111760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:06.111761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:06.111912 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:06.112151 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:06.112152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:06.112153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:06.112471 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:06.112621 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:06.112754 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 10:55:06.112892 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":100,"threads_started":1}
I20260929 10:55:06.114179 18765 catalog_manager.cc:1347] Generated new cluster ID: da4a938c788548bdb2001057c8e3e1c4
I20260929 10:55:06.114228 18765 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:55:06.125366 18765 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:55:06.125965 18765 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:55:06.133841 18765 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5072e103f2c4e39a27e638aa276acb7: Generated new TSK 0
I20260929 10:55:06.134167 18765 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.10.238.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.238.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.238.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.193.keytab.
Entry for principal oryx/127.10.238.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.193.keytab.
I20260929 10:55:06.164671 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.193:0
--local_ip_for_outbound_sockets=127.10.238.193
--webserver_interface=127.10.238.193
--webserver_port=0
--tserver_master_addrs=127.10.238.254:36571
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.193.keytab
--principal=oryx/127.10.238.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:41141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:55:06.249506 18776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:55:06.249724 18776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:55:06.249749 18776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:55:06.250921 18776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:55:06.250967 18776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:55:06.250988 18776 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 10:55:06.251046 18776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.193
I20260929 10:55:06.252555 18776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:41141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.193.keytab
--principal=oryx/127.10.238.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.238.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.238.254:36571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.193
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:55:06.252804 18776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:55:06.253065 18776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:55:06.258018 18782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:06.258054 18781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:06.258062 18784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:55:06.258251 18776 server_base.cc:1034] running on GCE node
I20260929 10:55:06.258469 18776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:55:06.258713 18776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:55:06.259876 18776 hybrid_clock.cc:648] HybridClock initialized: now 1790679306259864 us; error 30 us; skew 500 ppm
Sep 29 10:55:06 dist-test-slave-8nrt krb5kdc[18659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679306, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.238.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:55:06.262428 18776 init.cc:377] Logged in from keytab as oryx/127.10.238.193@KRBTEST.COM (short username oryx)
I20260929 10:55:06.262866 18776 webserver.cc:460] Webserver started at http://127.10.238.193:36073/ using document root <none> and password file <none>
I20260929 10:55:06.263098 18776 fs_manager.cc:359] Metadata directory not provided
I20260929 10:55:06.263166 18776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:55:06.263336 18776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:55:06.264411 18776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-0/data/instance:
uuid: "62f127242e214ef497866289454389c2"
format_stamp: "Formatted at 2026-09-29 10:55:06 on dist-test-slave-8nrt"
server_key: "6d9a1e783d3d9e1c024b75486956da3b"
server_key_iv: "55af78e1927f620b669e20d267690005"
server_key_version: "encryptionkey@0"
I20260929 10:55:06.264796 18776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-0/wal/instance:
uuid: "62f127242e214ef497866289454389c2"
format_stamp: "Formatted at 2026-09-29 10:55:06 on dist-test-slave-8nrt"
server_key: "6d9a1e783d3d9e1c024b75486956da3b"
server_key_iv: "55af78e1927f620b669e20d267690005"
server_key_version: "encryptionkey@0"
I20260929 10:55:06.266291 18776 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 10:55:06.267143 18791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:06.267374 18776 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:55:06.267490 18776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-0/wal
uuid: "62f127242e214ef497866289454389c2"
format_stamp: "Formatted at 2026-09-29 10:55:06 on dist-test-slave-8nrt"
server_key: "6d9a1e783d3d9e1c024b75486956da3b"
server_key_iv: "55af78e1927f620b669e20d267690005"
server_key_version: "encryptionkey@0"
I20260929 10:55:06.267570 18776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:55:06.278323 18776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:55:06.280457 18776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:55:06.280656 18776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:55:06.281204 18776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:55:06.281266 18776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:06.281332 18776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:55:06.281364 18776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:06.289134 18776 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.193:34399
I20260929 10:55:06.289213 18906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.193:34399 every 8 connection(s)
Sep 29 10:55:06 dist-test-slave-8nrt krb5kdc[18659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679306, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.238.193@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM
I20260929 10:55:06.289593 18776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
I20260929 10:55:06.290040 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 18776
I20260929 10:55:06.290172 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-0/wal/instance
I20260929 10:55:06.290464 11195 external_mini_cluster.cc:1428] Setting key 47b034521717b43628615f62437cf011
I20260929 10:55:06.293440 18753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:06.281709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:44279 (local address 127.10.238.254:36571)
0929 10:55:06.281781 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:06.281784 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:06.282081 (+ 297us) server_negotiation.cc:407] Connection header received
0929 10:55:06.282837 (+ 756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:06.282839 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:06.282853 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:06.282932 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:06.283895 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.284797 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.286592 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.286961 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.287037 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:06.290200 (+ 3163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:06.290202 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:06.290203 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:06.290212 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:06.291953 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:06.292357 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:06.292358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:06.292359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:06.292434 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:06.292694 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:06.292695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:06.292695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:06.292836 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:06.292920 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:06.293212 (+ 292us) server_negotiation.cc:299] Negotiation successful
0929 10:55:06.293347 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 10:55:06.293620 18800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:06.281686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:36571 (local address 127.10.238.193:44279)
0929 10:55:06.281900 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:06.281956 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:06.282705 (+ 749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:06.283018 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:06.283024 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:06.283453 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:06.283783 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:06.283791 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.284904 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.284905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:06.286485 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:06.286489 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.287072 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.287074 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:06.287252 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:06.288023 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:06.288035 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:06.290062 (+ 2027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:55:06.292043 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:06.292050 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:06.292052 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:06.292278 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:06.292503 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:06.292503 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:06.292504 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:06.292612 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:06.292918 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:55:06.292926 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:55:06.293074 (+ 148us) client_negotiation.cc:769] Sending connection context
0929 10:55:06.293190 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 10:55:06.293380 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":97,"threads_started":1}
I20260929 10:55:06.294107 18907 heartbeater.cc:347] Connected to a master server at 127.10.238.254:36571
I20260929 10:55:06.294209 18907 heartbeater.cc:464] Registering TS with master...
I20260929 10:55:06.294404 18907 heartbeater.cc:511] Master 127.10.238.254:36571 requested a full tablet report, sending...
I20260929 10:55:06.294977 18703 ts_manager.cc:194] Registered new tserver with Master: 62f127242e214ef497866289454389c2 (127.10.238.193:34399)
I20260929 10:55:06.295815 18703 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.238.193@KRBTEST.COM'} at 127.10.238.193:44279
WARNING: no policy specified for oryx/127.10.238.194@KRBTEST.COM; defaulting to no policy
I20260929 10:55:06.302382 18753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:06.296031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:36363 (local address 127.10.238.254:36571)
0929 10:55:06.296069 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:06.296071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:06.296081 (+ 10us) server_negotiation.cc:407] Connection header received
0929 10:55:06.296100 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:06.296101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:06.296111 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:06.296172 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:06.296637 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.297491 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.298253 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.298572 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.298642 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:06.299791 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:06.299793 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:06.299793 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:06.299799 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:06.301363 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:06.301672 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:06.301672 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:06.301673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:06.301732 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:06.301896 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:06.301896 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:06.301897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:06.302036 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:06.302108 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:06.302251 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 10:55:06.302316 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 10:55:06.302395 18914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:06.295287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:36571 (local address 127.10.238.193:36363)
0929 10:55:06.295439 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:06.295451 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:06.295505 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:06.296185 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:06.296187 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:06.296444 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:06.296522 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:06.296525 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.297621 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.297622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:06.298161 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:06.298162 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.298714 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.298715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:06.298798 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:06.299265 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:06.299270 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:06.299703 (+ 433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:55:06.301425 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:06.301426 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:06.301427 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:06.301602 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:06.301792 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:06.301792 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:06.301793 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:06.301839 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:06.302109 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:55:06.302110 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:55:06.302154 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 10:55:06.302225 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 10:55:06.302315 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":74,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.238.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.238.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.194.keytab.
Entry for principal oryx/127.10.238.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.194.keytab.
I20260929 10:55:06.320765 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.194:0
--local_ip_for_outbound_sockets=127.10.238.194
--webserver_interface=127.10.238.194
--webserver_port=0
--tserver_master_addrs=127.10.238.254:36571
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.194.keytab
--principal=oryx/127.10.238.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:41141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:55:06.402175 18916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:55:06.402412 18916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:55:06.402450 18916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:55:06.403604 18916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:55:06.403669 18916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:55:06.403703 18916 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 10:55:06.403751 18916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.194
I20260929 10:55:06.405221 18916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:41141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.194.keytab
--principal=oryx/127.10.238.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.238.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.238.254:36571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.194
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:55:06.405522 18916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:55:06.405781 18916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:55:06.410866 18921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:55:06.410904 18916 server_base.cc:1034] running on GCE node
W20260929 10:55:06.411008 18922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:06.410876 18924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:55:06.411360 18916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:55:06.411615 18916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:55:06.412798 18916 hybrid_clock.cc:648] HybridClock initialized: now 1790679306412780 us; error 40 us; skew 500 ppm
Sep 29 10:55:06 dist-test-slave-8nrt krb5kdc[18659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679306, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.238.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:55:06.415501 18916 init.cc:377] Logged in from keytab as oryx/127.10.238.194@KRBTEST.COM (short username oryx)
I20260929 10:55:06.416023 18916 webserver.cc:460] Webserver started at http://127.10.238.194:38171/ using document root <none> and password file <none>
I20260929 10:55:06.416252 18916 fs_manager.cc:359] Metadata directory not provided
I20260929 10:55:06.416332 18916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:55:06.416502 18916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:55:06.417639 18916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-1/data/instance:
uuid: "a033d9467d4d451586b0506fff8ebdf9"
format_stamp: "Formatted at 2026-09-29 10:55:06 on dist-test-slave-8nrt"
server_key: "adbb580c910452889a165d44cbddcb45"
server_key_iv: "aa64a1052ef4ee201c6b50c5203a267d"
server_key_version: "encryptionkey@0"
I20260929 10:55:06.418020 18916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-1/wal/instance:
uuid: "a033d9467d4d451586b0506fff8ebdf9"
format_stamp: "Formatted at 2026-09-29 10:55:06 on dist-test-slave-8nrt"
server_key: "adbb580c910452889a165d44cbddcb45"
server_key_iv: "aa64a1052ef4ee201c6b50c5203a267d"
server_key_version: "encryptionkey@0"
I20260929 10:55:06.419430 18916 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 10:55:06.420245 18931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:06.420416 18916 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:55:06.420523 18916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-1/wal
uuid: "a033d9467d4d451586b0506fff8ebdf9"
format_stamp: "Formatted at 2026-09-29 10:55:06 on dist-test-slave-8nrt"
server_key: "adbb580c910452889a165d44cbddcb45"
server_key_iv: "aa64a1052ef4ee201c6b50c5203a267d"
server_key_version: "encryptionkey@0"
I20260929 10:55:06.420599 18916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:55:06.446394 18916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:55:06.448899 18916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:55:06.449074 18916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:55:06.449767 18916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:55:06.449819 18916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:06.449857 18916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:55:06.449872 18916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 10:55:06 dist-test-slave-8nrt krb5kdc[18659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679306, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.238.194@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM
I20260929 10:55:06.457762 18916 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.194:35881
I20260929 10:55:06.457902 19046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.194:35881 every 8 connection(s)
I20260929 10:55:06.458184 18916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
I20260929 10:55:06.460340 18753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:06.450307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:56571 (local address 127.10.238.254:36571)
0929 10:55:06.450380 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:06.450384 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:06.450591 (+ 207us) server_negotiation.cc:407] Connection header received
0929 10:55:06.451170 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:06.451172 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:06.451186 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:06.451284 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:06.452346 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.453199 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.454517 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.454773 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.454859 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:06.457476 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:06.457478 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:06.457478 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:06.457486 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:06.459174 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:06.459575 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:06.459576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:06.459576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:06.459628 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:06.459802 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:06.459802 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:06.459802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:06.459923 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:06.459973 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:06.460213 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 10:55:06.460274 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 10:55:06.460593 18940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:06.450268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:36571 (local address 127.10.238.194:56571)
0929 10:55:06.450441 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:06.450487 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:06.451032 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:06.451383 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:06.451388 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:06.451831 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:06.452230 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:06.452240 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.453344 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.453345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:06.454337 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:06.454341 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.454904 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.454906 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:06.455051 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:06.455650 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:06.455658 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:06.457376 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:55:06.459268 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:06.459274 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:06.459276 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:06.459503 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:06.459669 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:06.459670 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:06.459670 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:06.459753 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:06.459983 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:55:06.459989 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:55:06.460103 (+ 114us) client_negotiation.cc:769] Sending connection context
0929 10:55:06.460187 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 10:55:06.460321 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":89,"threads_started":1}
I20260929 10:55:06.461177 19047 heartbeater.cc:347] Connected to a master server at 127.10.238.254:36571
I20260929 10:55:06.461300 19047 heartbeater.cc:464] Registering TS with master...
I20260929 10:55:06.461484 19047 heartbeater.cc:511] Master 127.10.238.254:36571 requested a full tablet report, sending...
I20260929 10:55:06.461939 18704 ts_manager.cc:194] Registered new tserver with Master: a033d9467d4d451586b0506fff8ebdf9 (127.10.238.194:35881)
I20260929 10:55:06.462472 18704 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.238.194@KRBTEST.COM'} at 127.10.238.194:56571
I20260929 10:55:06.466046 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 18916
I20260929 10:55:06.466171 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-1/wal/instance
I20260929 10:55:06.466428 11195 external_mini_cluster.cc:1428] Setting key 87917226bb2e78a2b03c776ee1f7e16f
I20260929 10:55:06.468070 18753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:06.462293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:39005 (local address 127.10.238.254:36571)
0929 10:55:06.462341 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:06.462342 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:06.462412 (+ 70us) server_negotiation.cc:407] Connection header received
0929 10:55:06.462473 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:06.462473 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:06.462481 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:06.462523 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:06.462924 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.463408 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.464109 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.464320 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.464403 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:06.465356 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:06.465358 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:06.465358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:06.465364 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:06.466904 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:06.467253 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:06.467254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:06.467255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:06.467310 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:06.467519 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:06.467519 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:06.467520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:06.467667 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:06.467740 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:06.467906 (+ 166us) server_negotiation.cc:299] Negotiation successful
0929 10:55:06.467980 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 10:55:06.468078 19053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:06.462208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:36571 (local address 127.10.238.194:39005)
0929 10:55:06.462350 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:06.462361 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:06.462419 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:06.462529 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:06.462530 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:06.462759 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:06.462838 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:06.462841 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.463503 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.463504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:06.464026 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:06.464027 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.464407 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.464408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:06.464477 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:06.464935 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:06.464940 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:06.465214 (+ 274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:55:06.467006 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:06.467007 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:06.467008 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:06.467180 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:06.467385 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:06.467386 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:06.467387 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:06.467444 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:06.467745 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:55:06.467746 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:55:06.467792 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 10:55:06.467878 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 10:55:06.467983 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":76,"threads_started":1}
WARNING: no policy specified for oryx/127.10.238.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.238.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.238.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.195.keytab.
Entry for principal oryx/127.10.238.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.195.keytab.
I20260929 10:55:06.492206 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.195:0
--local_ip_for_outbound_sockets=127.10.238.195
--webserver_interface=127.10.238.195
--webserver_port=0
--tserver_master_addrs=127.10.238.254:36571
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.195.keytab
--principal=oryx/127.10.238.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:41141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:55:06.574165 19056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:55:06.574364 19056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:55:06.574388 19056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:55:06.575474 19056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:55:06.575506 19056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:55:06.575526 19056 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 10:55:06.575542 19056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.195
I20260929 10:55:06.577098 19056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:41141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.195.keytab
--principal=oryx/127.10.238.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.238.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.238.254:36571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.195
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:55:06.577355 19056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:55:06.577541 19056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:55:06.582617 19064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:06.582621 19062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:06.582664 19061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:55:06.582969 19056 server_base.cc:1034] running on GCE node
I20260929 10:55:06.583181 19056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:55:06.583388 19056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:55:06.584537 19056 hybrid_clock.cc:648] HybridClock initialized: now 1790679306584507 us; error 47 us; skew 500 ppm
Sep 29 10:55:06 dist-test-slave-8nrt krb5kdc[18659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679306, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.238.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:55:06.587117 19056 init.cc:377] Logged in from keytab as oryx/127.10.238.195@KRBTEST.COM (short username oryx)
I20260929 10:55:06.587596 19056 webserver.cc:460] Webserver started at http://127.10.238.195:36351/ using document root <none> and password file <none>
I20260929 10:55:06.587839 19056 fs_manager.cc:359] Metadata directory not provided
I20260929 10:55:06.587883 19056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:55:06.588047 19056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:55:06.589076 19056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-2/data/instance:
uuid: "44a08b4b7d1c4435bee2560269289eb3"
format_stamp: "Formatted at 2026-09-29 10:55:06 on dist-test-slave-8nrt"
server_key: "60b7cbe7f6d4a095c49dae333bcd7a1d"
server_key_iv: "219b87ef3c92c88a7b20e422c8b36347"
server_key_version: "encryptionkey@0"
I20260929 10:55:06.589464 19056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-2/wal/instance:
uuid: "44a08b4b7d1c4435bee2560269289eb3"
format_stamp: "Formatted at 2026-09-29 10:55:06 on dist-test-slave-8nrt"
server_key: "60b7cbe7f6d4a095c49dae333bcd7a1d"
server_key_iv: "219b87ef3c92c88a7b20e422c8b36347"
server_key_version: "encryptionkey@0"
I20260929 10:55:06.590778 19056 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 10:55:06.591460 19071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:06.591655 19056 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 10:55:06.591754 19056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-2/wal
uuid: "44a08b4b7d1c4435bee2560269289eb3"
format_stamp: "Formatted at 2026-09-29 10:55:06 on dist-test-slave-8nrt"
server_key: "60b7cbe7f6d4a095c49dae333bcd7a1d"
server_key_iv: "219b87ef3c92c88a7b20e422c8b36347"
server_key_version: "encryptionkey@0"
I20260929 10:55:06.591830 19056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:55:06.608515 19056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:55:06.610760 19056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:55:06.610920 19056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:55:06.611589 19056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:55:06.611660 19056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:06.611692 19056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:55:06.611713 19056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 10:55:06 dist-test-slave-8nrt krb5kdc[18659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679306, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.238.195@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM
I20260929 10:55:06.619004 19056 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.195:38935
I20260929 10:55:06.619099 19186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.195:38935 every 8 connection(s)
I20260929 10:55:06.619431 19056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
I20260929 10:55:06.621230 18753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:06.611935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:44535 (local address 127.10.238.254:36571)
0929 10:55:06.611976 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:06.611978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:06.612178 (+ 200us) server_negotiation.cc:407] Connection header received
0929 10:55:06.612720 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:06.612721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:06.612733 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:06.612799 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:06.613618 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.614139 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.615195 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.615403 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.615492 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:06.618390 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:06.618392 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:06.618392 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:06.618401 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:06.620068 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:06.620426 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:06.620426 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:06.620427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:06.620480 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:06.620693 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:06.620693 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:06.620694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:06.620798 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:06.620849 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:06.621100 (+ 251us) server_negotiation.cc:299] Negotiation successful
0929 10:55:06.621167 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 10:55:06.621399 19080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:06.611928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:36571 (local address 127.10.238.195:44535)
0929 10:55:06.612061 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:06.612095 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:06.612616 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:06.612869 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:06.612873 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:06.613241 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:06.613521 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:06.613527 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.614237 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.614238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:06.615085 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:06.615087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.615504 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.615505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:06.615638 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:06.616153 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:06.616160 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:06.618287 (+ 2127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:55:06.620138 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:06.620143 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:06.620144 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:06.620357 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:06.620538 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:06.620539 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:06.620539 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:06.620639 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:06.620854 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:55:06.620859 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:55:06.620968 (+ 109us) client_negotiation.cc:769] Sending connection context
0929 10:55:06.621050 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 10:55:06.621168 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":74,"threads_started":1}
I20260929 10:55:06.622025 19187 heartbeater.cc:347] Connected to a master server at 127.10.238.254:36571
I20260929 10:55:06.622108 19187 heartbeater.cc:464] Registering TS with master...
I20260929 10:55:06.622301 19187 heartbeater.cc:511] Master 127.10.238.254:36571 requested a full tablet report, sending...
I20260929 10:55:06.622639 18704 ts_manager.cc:194] Registered new tserver with Master: 44a08b4b7d1c4435bee2560269289eb3 (127.10.238.195:38935)
I20260929 10:55:06.623034 18704 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.238.195@KRBTEST.COM'} at 127.10.238.195:44535
I20260929 10:55:06.627352 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 19056
I20260929 10:55:06.627471 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-2/wal/instance
I20260929 10:55:06.627688 11195 external_mini_cluster.cc:1428] Setting key 4a9de1cddcfe8abfeeb7841911e75037
I20260929 10:55:06.628620 11195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:55:06.628798 19193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:06.623034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:36571 (local address 127.10.238.195:36083)
0929 10:55:06.623175 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:06.623185 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:06.623250 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:06.623748 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:06.623748 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:06.623889 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:06.623932 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:06.623933 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.624620 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.624621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:06.625125 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:06.625126 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.625498 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.625499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:06.625551 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:06.625880 (+ 329us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:06.625884 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:06.626130 (+ 246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:55:06.627692 (+ 1562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:06.627693 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:06.627693 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:06.627923 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:06.628134 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:06.628135 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:06.628136 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:06.628203 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:06.628479 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:55:06.628480 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:55:06.628526 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 10:55:06.628613 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 10:55:06.628715 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":80,"threads_started":1}
I20260929 10:55:06.628811 18753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:06.623103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:36083 (local address 127.10.238.254:36571)
0929 10:55:06.623175 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:06.623177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:06.623228 (+ 51us) server_negotiation.cc:407] Connection header received
0929 10:55:06.623312 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:06.623313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:06.623322 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:06.623371 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:06.623996 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.624538 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.625189 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.625436 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.625486 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:06.626204 (+ 718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:06.626205 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:06.626205 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:06.626211 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:06.627618 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:06.628003 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:06.628003 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:06.628004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:06.628066 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:06.628278 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:06.628279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:06.628279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:06.628411 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:06.628475 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:06.628641 (+ 166us) server_negotiation.cc:299] Negotiation successful
0929 10:55:06.628720 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Sep 29 10:55:06 dist-test-slave-8nrt krb5kdc[18659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679306, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 10:55:06 dist-test-slave-8nrt krb5kdc[18659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:06 dist-test-slave-8nrt krb5kdc[18659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
W20260929 10:55:06.649456 18753 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:06.644228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33022 (local address 127.10.238.254:36571)
0929 10:55:06.644294 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:06.644296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:06.644313 (+ 17us) server_negotiation.cc:407] Connection header received
0929 10:55:06.644355 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:06.644356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:06.644364 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:06.644405 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:06.644930 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.645470 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.646185 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.646354 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.646399 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:06.649364 (+ 2965us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33022: BlockingRecv error: recv got EOF from 127.0.0.1:33022 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 29 10:55:06 dist-test-slave-8nrt krb5kdc[18659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679306, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM
I20260929 10:55:06.656873 18753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:06.650218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33028 (local address 127.10.238.254:36571)
0929 10:55:06.650289 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:06.650291 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:06.650379 (+ 88us) server_negotiation.cc:407] Connection header received
0929 10:55:06.650455 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:06.650456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:06.650466 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:06.650524 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:06.651004 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.651482 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.652177 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.652368 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.652442 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:06.654542 (+ 2100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:06.654544 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:06.654545 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:06.654552 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:06.655760 (+ 1208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:06.656091 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:06.656092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:06.656093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:06.656153 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:06.656357 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:06.656358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:06.656369 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:06.656483 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:06.656591 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:06.656691 (+ 100us) server_negotiation.cc:299] Negotiation successful
0929 10:55:06.656768 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 10:55:06.658324 18704 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33028:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:55:06.659865 18704 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:55:06.666378 19212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:06.662292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59466 (local address 127.10.238.193:34399)
0929 10:55:06.662511 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:06.662516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:06.662527 (+ 11us) server_negotiation.cc:407] Connection header received
0929 10:55:06.662568 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:06.662571 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:06.662589 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:06.662664 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:55:06.662977 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.663876 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.665359 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.665916 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.666037 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:06.666102 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:06.666242 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 10:55:06.666279 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":96,"threads_started":1}
I20260929 10:55:06.666368 19210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:06.662126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:34399 (local address 127.0.0.1:59466)
0929 10:55:06.662266 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:06.662321 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:06.662423 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:06.662667 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:06.662669 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:06.662671 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:55:06.662740 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:06.662742 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.663963 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.663965 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:06.665249 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:06.665283 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.666022 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.666023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:06.666103 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:06.666119 (+ 16us) client_negotiation.cc:769] Sending connection context
0929 10:55:06.666222 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 10:55:06.666279 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":59,"threads_started":1}
I20260929 10:55:06.666731 19211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:06.662260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35296 (local address 127.10.238.194:35881)
0929 10:55:06.662625 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:06.662630 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:06.662642 (+ 12us) server_negotiation.cc:407] Connection header received
0929 10:55:06.662690 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:06.662693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:06.662708 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:06.662804 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:55:06.663028 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.664125 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.665387 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.666155 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.666267 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:06.666322 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:06.666585 (+ 263us) server_negotiation.cc:299] Negotiation successful
0929 10:55:06.666633 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":96,"threads_started":1}
I20260929 10:55:06.666968 19214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:06.662574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53794 (local address 127.10.238.195:38935)
0929 10:55:06.662772 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:06.662776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:06.662953 (+ 177us) server_negotiation.cc:407] Connection header received
0929 10:55:06.663007 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:06.663009 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:06.663027 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:06.663092 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:55:06.663307 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.664323 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.665482 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.666183 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.666267 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:06.666322 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:06.666839 (+ 517us) server_negotiation.cc:299] Negotiation successful
0929 10:55:06.666884 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":94,"threads_started":1}
I20260929 10:55:06.667132 19209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:06.662348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.195:38935 (local address 127.0.0.1:53794)
0929 10:55:06.662758 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:06.662768 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:06.662836 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:06.663165 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:06.663166 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:06.663167 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:55:06.663210 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:06.663211 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.664409 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.664410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:06.665398 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:06.665401 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.666363 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.666363 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:06.666446 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:06.666451 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 10:55:06.666501 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 10:55:06.666540 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":103,"thread_start_us":80,"threads_started":1}
I20260929 10:55:06.667301 18841 tablet_service.cc:1467] Processing CreateTablet for tablet 292f03d610f34c5e8c379a9b39ddf82d (DEFAULT_TABLE table=test-table [id=1d33c84f899a4fad82d8f319cc07a0e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:55:06.667721 18841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 292f03d610f34c5e8c379a9b39ddf82d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:55:06.667762 19121 tablet_service.cc:1467] Processing CreateTablet for tablet 292f03d610f34c5e8c379a9b39ddf82d (DEFAULT_TABLE table=test-table [id=1d33c84f899a4fad82d8f319cc07a0e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:55:06.668139 19121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 292f03d610f34c5e8c379a9b39ddf82d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:55:06.669399 19213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:06.662137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.194:35881 (local address 127.0.0.1:35296)
0929 10:55:06.662472 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:06.662483 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:06.662539 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:06.662879 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:06.662880 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:06.662882 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:55:06.662935 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:06.662937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.664211 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.664212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:06.665300 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:06.665302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.666361 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.666362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:06.666446 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:06.666450 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 10:55:06.666501 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 10:55:06.666541 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":3,"thread_start_us":76,"threads_started":1}
I20260929 10:55:06.671320 19216 tablet_bootstrap.cc:492] T 292f03d610f34c5e8c379a9b39ddf82d P 62f127242e214ef497866289454389c2: Bootstrap starting.
I20260929 10:55:06.671391 19215 tablet_bootstrap.cc:492] T 292f03d610f34c5e8c379a9b39ddf82d P 44a08b4b7d1c4435bee2560269289eb3: Bootstrap starting.
I20260929 10:55:06.671995 19216 tablet_bootstrap.cc:654] T 292f03d610f34c5e8c379a9b39ddf82d P 62f127242e214ef497866289454389c2: Neither blocks nor log segments found. Creating new log.
I20260929 10:55:06.672106 19215 tablet_bootstrap.cc:654] T 292f03d610f34c5e8c379a9b39ddf82d P 44a08b4b7d1c4435bee2560269289eb3: Neither blocks nor log segments found. Creating new log.
I20260929 10:55:06.672338 19216 log.cc:824] T 292f03d610f34c5e8c379a9b39ddf82d P 62f127242e214ef497866289454389c2: Log is configured to *not* fsync() on all Append() calls
I20260929 10:55:06.672221 18981 tablet_service.cc:1467] Processing CreateTablet for tablet 292f03d610f34c5e8c379a9b39ddf82d (DEFAULT_TABLE table=test-table [id=1d33c84f899a4fad82d8f319cc07a0e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:55:06.672456 19215 log.cc:824] T 292f03d610f34c5e8c379a9b39ddf82d P 44a08b4b7d1c4435bee2560269289eb3: Log is configured to *not* fsync() on all Append() calls
I20260929 10:55:06.672603 18981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 292f03d610f34c5e8c379a9b39ddf82d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:55:06.673250 19215 tablet_bootstrap.cc:492] T 292f03d610f34c5e8c379a9b39ddf82d P 44a08b4b7d1c4435bee2560269289eb3: No bootstrap required, opened a new log
I20260929 10:55:06.673370 19215 ts_tablet_manager.cc:1397] T 292f03d610f34c5e8c379a9b39ddf82d P 44a08b4b7d1c4435bee2560269289eb3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 10:55:06.673959 19216 tablet_bootstrap.cc:492] T 292f03d610f34c5e8c379a9b39ddf82d P 62f127242e214ef497866289454389c2: No bootstrap required, opened a new log
I20260929 10:55:06.674043 19216 ts_tablet_manager.cc:1397] T 292f03d610f34c5e8c379a9b39ddf82d P 62f127242e214ef497866289454389c2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 10:55:06.675436 19216 raft_consensus.cc:348] T 292f03d610f34c5e8c379a9b39ddf82d P 62f127242e214ef497866289454389c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } }
I20260929 10:55:06.675594 19216 raft_consensus.cc:374] T 292f03d610f34c5e8c379a9b39ddf82d P 62f127242e214ef497866289454389c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:55:06.675639 19216 raft_consensus.cc:729] T 292f03d610f34c5e8c379a9b39ddf82d P 62f127242e214ef497866289454389c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62f127242e214ef497866289454389c2, State: Initialized, Role: FOLLOWER
I20260929 10:55:06.675544 19215 raft_consensus.cc:348] T 292f03d610f34c5e8c379a9b39ddf82d P 44a08b4b7d1c4435bee2560269289eb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } }
I20260929 10:55:06.675685 19215 raft_consensus.cc:374] T 292f03d610f34c5e8c379a9b39ddf82d P 44a08b4b7d1c4435bee2560269289eb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:55:06.675726 19215 raft_consensus.cc:729] T 292f03d610f34c5e8c379a9b39ddf82d P 44a08b4b7d1c4435bee2560269289eb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44a08b4b7d1c4435bee2560269289eb3, State: Initialized, Role: FOLLOWER
I20260929 10:55:06.675751 19216 consensus_queue.cc:260] T 292f03d610f34c5e8c379a9b39ddf82d P 62f127242e214ef497866289454389c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } }
I20260929 10:55:06.675835 19215 consensus_queue.cc:260] T 292f03d610f34c5e8c379a9b39ddf82d P 44a08b4b7d1c4435bee2560269289eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } }
I20260929 10:55:06.676036 19216 ts_tablet_manager.cc:1428] T 292f03d610f34c5e8c379a9b39ddf82d P 62f127242e214ef497866289454389c2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 10:55:06.676090 19215 ts_tablet_manager.cc:1428] T 292f03d610f34c5e8c379a9b39ddf82d P 44a08b4b7d1c4435bee2560269289eb3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 10:55:06.676178 18907 heartbeater.cc:503] Master 127.10.238.254:36571 was elected leader, sending a full tablet report...
I20260929 10:55:06.676244 19187 heartbeater.cc:503] Master 127.10.238.254:36571 was elected leader, sending a full tablet report...
I20260929 10:55:06.677101 19221 tablet_bootstrap.cc:492] T 292f03d610f34c5e8c379a9b39ddf82d P a033d9467d4d451586b0506fff8ebdf9: Bootstrap starting.
I20260929 10:55:06.677747 19221 tablet_bootstrap.cc:654] T 292f03d610f34c5e8c379a9b39ddf82d P a033d9467d4d451586b0506fff8ebdf9: Neither blocks nor log segments found. Creating new log.
I20260929 10:55:06.678066 19221 log.cc:824] T 292f03d610f34c5e8c379a9b39ddf82d P a033d9467d4d451586b0506fff8ebdf9: Log is configured to *not* fsync() on all Append() calls
I20260929 10:55:06.678736 19221 tablet_bootstrap.cc:492] T 292f03d610f34c5e8c379a9b39ddf82d P a033d9467d4d451586b0506fff8ebdf9: No bootstrap required, opened a new log
I20260929 10:55:06.678821 19221 ts_tablet_manager.cc:1397] T 292f03d610f34c5e8c379a9b39ddf82d P a033d9467d4d451586b0506fff8ebdf9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 10:55:06.680248 19221 raft_consensus.cc:348] T 292f03d610f34c5e8c379a9b39ddf82d P a033d9467d4d451586b0506fff8ebdf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } }
I20260929 10:55:06.680406 19221 raft_consensus.cc:374] T 292f03d610f34c5e8c379a9b39ddf82d P a033d9467d4d451586b0506fff8ebdf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:55:06.680454 19221 raft_consensus.cc:729] T 292f03d610f34c5e8c379a9b39ddf82d P a033d9467d4d451586b0506fff8ebdf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a033d9467d4d451586b0506fff8ebdf9, State: Initialized, Role: FOLLOWER
I20260929 10:55:06.680613 19221 consensus_queue.cc:260] T 292f03d610f34c5e8c379a9b39ddf82d P a033d9467d4d451586b0506fff8ebdf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } }
I20260929 10:55:06.680866 19221 ts_tablet_manager.cc:1428] T 292f03d610f34c5e8c379a9b39ddf82d P a033d9467d4d451586b0506fff8ebdf9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 10:55:06.680910 19047 heartbeater.cc:503] Master 127.10.238.254:36571 was elected leader, sending a full tablet report...
W20260929 10:55:06.709091 19048 tablet.cc:2369] T 292f03d610f34c5e8c379a9b39ddf82d P a033d9467d4d451586b0506fff8ebdf9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:55:06.741300 19223 raft_consensus.cc:482] T 292f03d610f34c5e8c379a9b39ddf82d P a033d9467d4d451586b0506fff8ebdf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:55:06.741474 19223 raft_consensus.cc:504] T 292f03d610f34c5e8c379a9b39ddf82d P a033d9467d4d451586b0506fff8ebdf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } }
I20260929 10:55:06.741962 19223 leader_election.cc:288] T 292f03d610f34c5e8c379a9b39ddf82d P a033d9467d4d451586b0506fff8ebdf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62f127242e214ef497866289454389c2 (127.10.238.193:34399), 44a08b4b7d1c4435bee2560269289eb3 (127.10.238.195:38935)
I20260929 10:55:06.745536 19214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:06.742391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:52381 (local address 127.10.238.195:38935)
0929 10:55:06.742443 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:06.742445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:06.742511 (+ 66us) server_negotiation.cc:407] Connection header received
0929 10:55:06.742590 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:06.742591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:06.742601 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:06.742658 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:55:06.742864 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.743571 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.744569 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.745094 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.745175 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:06.745183 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:06.745408 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 10:55:06.745452 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 10:55:06.746045 19212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:06.742234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:48463 (local address 127.10.238.193:34399)
0929 10:55:06.742280 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:06.742282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:06.742296 (+ 14us) server_negotiation.cc:407] Connection header received
0929 10:55:06.742332 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:06.742333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:06.742343 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:06.742406 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:55:06.742623 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.743364 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.744877 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.745436 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.745499 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:06.745507 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:06.745942 (+ 435us) server_negotiation.cc:299] Negotiation successful
0929 10:55:06.745982 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 10:55:06.746042 19224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:06.742144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.195:38935 (local address 127.10.238.194:52381)
0929 10:55:06.742451 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:06.742461 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:06.742522 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:06.742685 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:06.742686 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:06.742688 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:55:06.742780 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:06.742782 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.743673 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.743674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:06.744457 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:06.744460 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.745189 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.745190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:06.745312 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:06.745318 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 10:55:06.745398 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 10:55:06.745453 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":14,"thread_start_us":105,"threads_started":1}
I20260929 10:55:06.746068 18940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:06.742143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:34399 (local address 127.10.238.194:48463)
0929 10:55:06.742209 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:06.742222 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:06.742272 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:06.742471 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:06.742473 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:06.742474 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:55:06.742544 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:06.742546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.743468 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.743469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:06.744772 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:06.744775 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.745539 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.745540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:06.745856 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:06.745860 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 10:55:06.745935 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 10:55:06.745980 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 10:55:06.746471 19140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292f03d610f34c5e8c379a9b39ddf82d" candidate_uuid: "a033d9467d4d451586b0506fff8ebdf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44a08b4b7d1c4435bee2560269289eb3" is_pre_election: true
I20260929 10:55:06.746656 19140 raft_consensus.cc:2454] T 292f03d610f34c5e8c379a9b39ddf82d P 44a08b4b7d1c4435bee2560269289eb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a033d9467d4d451586b0506fff8ebdf9 in term 0.
I20260929 10:55:06.746769 18861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292f03d610f34c5e8c379a9b39ddf82d" candidate_uuid: "a033d9467d4d451586b0506fff8ebdf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62f127242e214ef497866289454389c2" is_pre_election: true
I20260929 10:55:06.746945 18861 raft_consensus.cc:2454] T 292f03d610f34c5e8c379a9b39ddf82d P 62f127242e214ef497866289454389c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a033d9467d4d451586b0506fff8ebdf9 in term 0.
I20260929 10:55:06.747223 18933 leader_election.cc:302] T 292f03d610f34c5e8c379a9b39ddf82d P a033d9467d4d451586b0506fff8ebdf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62f127242e214ef497866289454389c2, a033d9467d4d451586b0506fff8ebdf9; no voters:
I20260929 10:55:06.747421 19223 raft_consensus.cc:2781] T 292f03d610f34c5e8c379a9b39ddf82d P a033d9467d4d451586b0506fff8ebdf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:55:06.747509 19223 raft_consensus.cc:482] T 292f03d610f34c5e8c379a9b39ddf82d P a033d9467d4d451586b0506fff8ebdf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:55:06.747542 19223 raft_consensus.cc:3037] T 292f03d610f34c5e8c379a9b39ddf82d P a033d9467d4d451586b0506fff8ebdf9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:55:06.748356 19223 raft_consensus.cc:504] T 292f03d610f34c5e8c379a9b39ddf82d P a033d9467d4d451586b0506fff8ebdf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } }
I20260929 10:55:06.748749 19223 leader_election.cc:288] T 292f03d610f34c5e8c379a9b39ddf82d P a033d9467d4d451586b0506fff8ebdf9 [CANDIDATE]: Term 1 election: Requested vote from peers 62f127242e214ef497866289454389c2 (127.10.238.193:34399), 44a08b4b7d1c4435bee2560269289eb3 (127.10.238.195:38935)
I20260929 10:55:06.748765 18861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292f03d610f34c5e8c379a9b39ddf82d" candidate_uuid: "a033d9467d4d451586b0506fff8ebdf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62f127242e214ef497866289454389c2"
I20260929 10:55:06.748783 19140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292f03d610f34c5e8c379a9b39ddf82d" candidate_uuid: "a033d9467d4d451586b0506fff8ebdf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44a08b4b7d1c4435bee2560269289eb3"
I20260929 10:55:06.748842 18861 raft_consensus.cc:3037] T 292f03d610f34c5e8c379a9b39ddf82d P 62f127242e214ef497866289454389c2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:55:06.748893 19140 raft_consensus.cc:3037] T 292f03d610f34c5e8c379a9b39ddf82d P 44a08b4b7d1c4435bee2560269289eb3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:55:06.749734 18861 raft_consensus.cc:2454] T 292f03d610f34c5e8c379a9b39ddf82d P 62f127242e214ef497866289454389c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a033d9467d4d451586b0506fff8ebdf9 in term 1.
I20260929 10:55:06.749931 18933 leader_election.cc:302] T 292f03d610f34c5e8c379a9b39ddf82d P a033d9467d4d451586b0506fff8ebdf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62f127242e214ef497866289454389c2, a033d9467d4d451586b0506fff8ebdf9; no voters:
I20260929 10:55:06.750013 19223 raft_consensus.cc:2781] T 292f03d610f34c5e8c379a9b39ddf82d P a033d9467d4d451586b0506fff8ebdf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:55:06.750005 19140 raft_consensus.cc:2454] T 292f03d610f34c5e8c379a9b39ddf82d P 44a08b4b7d1c4435bee2560269289eb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a033d9467d4d451586b0506fff8ebdf9 in term 1.
I20260929 10:55:06.750288 19223 raft_consensus.cc:686] T 292f03d610f34c5e8c379a9b39ddf82d P a033d9467d4d451586b0506fff8ebdf9 [term 1 LEADER]: Becoming Leader. State: Replica: a033d9467d4d451586b0506fff8ebdf9, State: Running, Role: LEADER
I20260929 10:55:06.750394 19223 consensus_queue.cc:237] T 292f03d610f34c5e8c379a9b39ddf82d P a033d9467d4d451586b0506fff8ebdf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } }
I20260929 10:55:06.751371 18703 catalog_manager.cc:5495] T 292f03d610f34c5e8c379a9b39ddf82d P a033d9467d4d451586b0506fff8ebdf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a033d9467d4d451586b0506fff8ebdf9 (127.10.238.194). New cstate: current_term: 1 leader_uuid: "a033d9467d4d451586b0506fff8ebdf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } health_report { overall_health: UNKNOWN } } }
I20260929 10:55:06.758136 19211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:06.755168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35308 (local address 127.10.238.194:35881)
0929 10:55:06.755208 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:06.755210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:06.755223 (+ 13us) server_negotiation.cc:407] Connection header received
0929 10:55:06.755314 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:06.755314 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:06.755329 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:06.755381 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:55:06.755516 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.755968 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.757355 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.757557 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.757618 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:06.757818 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:55:06.757914 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:55:06.757973 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:06.758040 (+ 67us) server_negotiation.cc:299] Negotiation successful
0929 10:55:06.758058 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 10:55:06.759904 18861 raft_consensus.cc:1261] T 292f03d610f34c5e8c379a9b39ddf82d P 62f127242e214ef497866289454389c2 [term 1 FOLLOWER]: Refusing update from remote peer a033d9467d4d451586b0506fff8ebdf9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:55:06.759922 19140 raft_consensus.cc:1261] T 292f03d610f34c5e8c379a9b39ddf82d P 44a08b4b7d1c4435bee2560269289eb3 [term 1 FOLLOWER]: Refusing update from remote peer a033d9467d4d451586b0506fff8ebdf9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:55:06.760288 19225 consensus_queue.cc:1035] T 292f03d610f34c5e8c379a9b39ddf82d P a033d9467d4d451586b0506fff8ebdf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:55:06.760429 19223 consensus_queue.cc:1035] T 292f03d610f34c5e8c379a9b39ddf82d P a033d9467d4d451586b0506fff8ebdf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:55:06.763585 19230 mvcc.cc:205] Tried to move back new op lower bound from 7334622440485789696 to 7334622440450301952. Current Snapshot: MvccSnapshot[applied={T|T < 7334622440485789696}]
I20260929 10:55:06.765292 18704 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:55:06.765386 18704 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:55:06.765383 19229 mvcc.cc:205] Tried to move back new op lower bound from 7334622440485789696 to 7334622440450301952. Current Snapshot: MvccSnapshot[applied={T|T < 7334622440485789696 or (T in {7334622440485789696})}]
I20260929 10:55:06.766181 18704 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f5072e103f2c4e39a27e638aa276acb7: Sending DeleteTablet for 3 replicas of tablet 292f03d610f34c5e8c379a9b39ddf82d
I20260929 10:55:06.766587 18981 tablet_service.cc:1514] Processing DeleteTablet for tablet 292f03d610f34c5e8c379a9b39ddf82d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:55:06 UTC) from {username='oryx', principal='oryx/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:35296
I20260929 10:55:06.766700 18704 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33028:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:55:06.766839 18704 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:55:06.766867 19121 tablet_service.cc:1514] Processing DeleteTablet for tablet 292f03d610f34c5e8c379a9b39ddf82d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:55:06 UTC) from {username='oryx', principal='oryx/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:53794
I20260929 10:55:06.767220 18841 tablet_service.cc:1514] Processing DeleteTablet for tablet 292f03d610f34c5e8c379a9b39ddf82d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:55:06 UTC) from {username='oryx', principal='oryx/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:59466
I20260929 10:55:06.767413 19239 tablet_replica.cc:330] T 292f03d610f34c5e8c379a9b39ddf82d P 62f127242e214ef497866289454389c2: stopping tablet replica
I20260929 10:55:06.767557 19239 raft_consensus.cc:2229] T 292f03d610f34c5e8c379a9b39ddf82d P 62f127242e214ef497866289454389c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:55:06.767729 19239 raft_consensus.cc:2258] T 292f03d610f34c5e8c379a9b39ddf82d P 62f127242e214ef497866289454389c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:55:06.768064 18841 tablet_service.cc:1467] Processing CreateTablet for tablet 333c3468bd974ece948b9f9dafafec15 (DEFAULT_TABLE table=test-table [id=48e4772c72614fd9937748b936efa2ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:55:06.768144 18981 tablet_service.cc:1467] Processing CreateTablet for tablet 333c3468bd974ece948b9f9dafafec15 (DEFAULT_TABLE table=test-table [id=48e4772c72614fd9937748b936efa2ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:55:06.768200 18841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 333c3468bd974ece948b9f9dafafec15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:55:06.768262 18981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 333c3468bd974ece948b9f9dafafec15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:55:06.769413 19221 tablet_bootstrap.cc:492] T 333c3468bd974ece948b9f9dafafec15 P a033d9467d4d451586b0506fff8ebdf9: Bootstrap starting.
I20260929 10:55:06.769845 19221 tablet_bootstrap.cc:654] T 333c3468bd974ece948b9f9dafafec15 P a033d9467d4d451586b0506fff8ebdf9: Neither blocks nor log segments found. Creating new log.
I20260929 10:55:06.770242 19216 tablet_bootstrap.cc:492] T 333c3468bd974ece948b9f9dafafec15 P 62f127242e214ef497866289454389c2: Bootstrap starting.
I20260929 10:55:06.770303 19121 tablet_service.cc:1467] Processing CreateTablet for tablet 333c3468bd974ece948b9f9dafafec15 (DEFAULT_TABLE table=test-table [id=48e4772c72614fd9937748b936efa2ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:55:06.770427 19121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 333c3468bd974ece948b9f9dafafec15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:55:06.770679 19216 tablet_bootstrap.cc:654] T 333c3468bd974ece948b9f9dafafec15 P 62f127242e214ef497866289454389c2: Neither blocks nor log segments found. Creating new log.
I20260929 10:55:06.771304 19216 tablet_bootstrap.cc:492] T 333c3468bd974ece948b9f9dafafec15 P 62f127242e214ef497866289454389c2: No bootstrap required, opened a new log
I20260929 10:55:06.771358 19216 ts_tablet_manager.cc:1397] T 333c3468bd974ece948b9f9dafafec15 P 62f127242e214ef497866289454389c2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 10:55:06.771523 19216 raft_consensus.cc:348] T 333c3468bd974ece948b9f9dafafec15 P 62f127242e214ef497866289454389c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } }
I20260929 10:55:06.771596 19216 raft_consensus.cc:374] T 333c3468bd974ece948b9f9dafafec15 P 62f127242e214ef497866289454389c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:55:06.771617 19216 raft_consensus.cc:729] T 333c3468bd974ece948b9f9dafafec15 P 62f127242e214ef497866289454389c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62f127242e214ef497866289454389c2, State: Initialized, Role: FOLLOWER
I20260929 10:55:06.771656 19216 consensus_queue.cc:260] T 333c3468bd974ece948b9f9dafafec15 P 62f127242e214ef497866289454389c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } }
I20260929 10:55:06.771746 19216 ts_tablet_manager.cc:1428] T 333c3468bd974ece948b9f9dafafec15 P 62f127242e214ef497866289454389c2: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 10:55:06.771943 19215 tablet_bootstrap.cc:492] T 333c3468bd974ece948b9f9dafafec15 P 44a08b4b7d1c4435bee2560269289eb3: Bootstrap starting.
I20260929 10:55:06.772293 19239 ts_tablet_manager.cc:1905] T 292f03d610f34c5e8c379a9b39ddf82d P 62f127242e214ef497866289454389c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:55:06.772382 19215 tablet_bootstrap.cc:654] T 333c3468bd974ece948b9f9dafafec15 P 44a08b4b7d1c4435bee2560269289eb3: Neither blocks nor log segments found. Creating new log.
I20260929 10:55:06.773363 19237 tablet_replica.cc:330] T 292f03d610f34c5e8c379a9b39ddf82d P a033d9467d4d451586b0506fff8ebdf9: stopping tablet replica
I20260929 10:55:06.773520 19237 raft_consensus.cc:2229] T 292f03d610f34c5e8c379a9b39ddf82d P a033d9467d4d451586b0506fff8ebdf9 [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:55:06.773672 19237 raft_consensus.cc:2258] T 292f03d610f34c5e8c379a9b39ddf82d P a033d9467d4d451586b0506fff8ebdf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:55:06.773813 19238 tablet_replica.cc:330] T 292f03d610f34c5e8c379a9b39ddf82d P 44a08b4b7d1c4435bee2560269289eb3: stopping tablet replica
I20260929 10:55:06.774034 19238 raft_consensus.cc:2229] T 292f03d610f34c5e8c379a9b39ddf82d P 44a08b4b7d1c4435bee2560269289eb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:55:06.774226 19238 raft_consensus.cc:2258] T 292f03d610f34c5e8c379a9b39ddf82d P 44a08b4b7d1c4435bee2560269289eb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:55:06.774227 19239 ts_tablet_manager.cc:1918] T 292f03d610f34c5e8c379a9b39ddf82d P 62f127242e214ef497866289454389c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:55:06.774304 19239 log.cc:1196] T 292f03d610f34c5e8c379a9b39ddf82d P 62f127242e214ef497866289454389c2: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-0/wal/wals/292f03d610f34c5e8c379a9b39ddf82d
I20260929 10:55:06.774619 19239 ts_tablet_manager.cc:1939] T 292f03d610f34c5e8c379a9b39ddf82d P 62f127242e214ef497866289454389c2: Deleting consensus metadata
I20260929 10:55:06.774866 19215 tablet_bootstrap.cc:492] T 333c3468bd974ece948b9f9dafafec15 P 44a08b4b7d1c4435bee2560269289eb3: No bootstrap required, opened a new log
I20260929 10:55:06.774935 19215 ts_tablet_manager.cc:1397] T 333c3468bd974ece948b9f9dafafec15 P 44a08b4b7d1c4435bee2560269289eb3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 10:55:06.775060 18689 catalog_manager.cc:4841] TS 62f127242e214ef497866289454389c2 (127.10.238.193:34399): tablet 292f03d610f34c5e8c379a9b39ddf82d (table test-table [id=1d33c84f899a4fad82d8f319cc07a0e9]) successfully deleted
I20260929 10:55:06.775089 19215 raft_consensus.cc:348] T 333c3468bd974ece948b9f9dafafec15 P 44a08b4b7d1c4435bee2560269289eb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } }
I20260929 10:55:06.775161 19215 raft_consensus.cc:374] T 333c3468bd974ece948b9f9dafafec15 P 44a08b4b7d1c4435bee2560269289eb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:55:06.775190 19215 raft_consensus.cc:729] T 333c3468bd974ece948b9f9dafafec15 P 44a08b4b7d1c4435bee2560269289eb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44a08b4b7d1c4435bee2560269289eb3, State: Initialized, Role: FOLLOWER
I20260929 10:55:06.775239 19215 consensus_queue.cc:260] T 333c3468bd974ece948b9f9dafafec15 P 44a08b4b7d1c4435bee2560269289eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } }
I20260929 10:55:06.775321 19215 ts_tablet_manager.cc:1428] T 333c3468bd974ece948b9f9dafafec15 P 44a08b4b7d1c4435bee2560269289eb3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:06.774866 19238 ts_tablet_manager.cc:1905] T 292f03d610f34c5e8c379a9b39ddf82d P 44a08b4b7d1c4435bee2560269289eb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:55:06.775866 19237 ts_tablet_manager.cc:1905] T 292f03d610f34c5e8c379a9b39ddf82d P a033d9467d4d451586b0506fff8ebdf9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:55:06.777088 19238 ts_tablet_manager.cc:1918] T 292f03d610f34c5e8c379a9b39ddf82d P 44a08b4b7d1c4435bee2560269289eb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:55:06.777221 19238 log.cc:1196] T 292f03d610f34c5e8c379a9b39ddf82d P 44a08b4b7d1c4435bee2560269289eb3: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-2/wal/wals/292f03d610f34c5e8c379a9b39ddf82d
I20260929 10:55:06.777602 19238 ts_tablet_manager.cc:1939] T 292f03d610f34c5e8c379a9b39ddf82d P 44a08b4b7d1c4435bee2560269289eb3: Deleting consensus metadata
I20260929 10:55:06.777812 19237 ts_tablet_manager.cc:1918] T 292f03d610f34c5e8c379a9b39ddf82d P a033d9467d4d451586b0506fff8ebdf9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:55:06.777901 19237 log.cc:1196] T 292f03d610f34c5e8c379a9b39ddf82d P a033d9467d4d451586b0506fff8ebdf9: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-1/wal/wals/292f03d610f34c5e8c379a9b39ddf82d
I20260929 10:55:06.778208 18690 catalog_manager.cc:4841] TS 44a08b4b7d1c4435bee2560269289eb3 (127.10.238.195:38935): tablet 292f03d610f34c5e8c379a9b39ddf82d (table test-table [id=1d33c84f899a4fad82d8f319cc07a0e9]) successfully deleted
I20260929 10:55:06.778439 19237 ts_tablet_manager.cc:1939] T 292f03d610f34c5e8c379a9b39ddf82d P a033d9467d4d451586b0506fff8ebdf9: Deleting consensus metadata
I20260929 10:55:06.778883 18690 catalog_manager.cc:4841] TS a033d9467d4d451586b0506fff8ebdf9 (127.10.238.194:35881): tablet 292f03d610f34c5e8c379a9b39ddf82d (table test-table [id=1d33c84f899a4fad82d8f319cc07a0e9]) successfully deleted
I20260929 10:55:06.779286 19221 tablet_bootstrap.cc:492] T 333c3468bd974ece948b9f9dafafec15 P a033d9467d4d451586b0506fff8ebdf9: No bootstrap required, opened a new log
I20260929 10:55:06.779338 19221 ts_tablet_manager.cc:1397] T 333c3468bd974ece948b9f9dafafec15 P a033d9467d4d451586b0506fff8ebdf9: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20260929 10:55:06.779495 19221 raft_consensus.cc:348] T 333c3468bd974ece948b9f9dafafec15 P a033d9467d4d451586b0506fff8ebdf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } }
I20260929 10:55:06.779570 19221 raft_consensus.cc:374] T 333c3468bd974ece948b9f9dafafec15 P a033d9467d4d451586b0506fff8ebdf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:55:06.779592 19221 raft_consensus.cc:729] T 333c3468bd974ece948b9f9dafafec15 P a033d9467d4d451586b0506fff8ebdf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a033d9467d4d451586b0506fff8ebdf9, State: Initialized, Role: FOLLOWER
I20260929 10:55:06.779637 19221 consensus_queue.cc:260] T 333c3468bd974ece948b9f9dafafec15 P a033d9467d4d451586b0506fff8ebdf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } }
I20260929 10:55:06.779723 19221 ts_tablet_manager.cc:1428] T 333c3468bd974ece948b9f9dafafec15 P a033d9467d4d451586b0506fff8ebdf9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260929 10:55:06.791152 18908 tablet.cc:2369] T 333c3468bd974ece948b9f9dafafec15 P 62f127242e214ef497866289454389c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 10:55:06.870071 19188 tablet.cc:2369] T 333c3468bd974ece948b9f9dafafec15 P 44a08b4b7d1c4435bee2560269289eb3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:55:06.931017 19220 raft_consensus.cc:482] T 333c3468bd974ece948b9f9dafafec15 P 44a08b4b7d1c4435bee2560269289eb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:55:06.931116 19220 raft_consensus.cc:504] T 333c3468bd974ece948b9f9dafafec15 P 44a08b4b7d1c4435bee2560269289eb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } }
I20260929 10:55:06.931639 19220 leader_election.cc:288] T 333c3468bd974ece948b9f9dafafec15 P 44a08b4b7d1c4435bee2560269289eb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62f127242e214ef497866289454389c2 (127.10.238.193:34399), a033d9467d4d451586b0506fff8ebdf9 (127.10.238.194:35881)
I20260929 10:55:06.934744 19211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:06.931947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:48997 (local address 127.10.238.194:35881)
0929 10:55:06.932006 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:06.932008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:06.932020 (+ 12us) server_negotiation.cc:407] Connection header received
0929 10:55:06.932049 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:06.932050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:06.932061 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:06.932115 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:55:06.932394 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.932878 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.933992 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.934367 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.934469 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:06.934479 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:06.934610 (+ 131us) server_negotiation.cc:299] Negotiation successful
0929 10:55:06.934677 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260929 10:55:06.934744 19243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:06.931798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.194:35881 (local address 127.10.238.195:48997)
0929 10:55:06.931927 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:06.931938 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:06.931990 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:06.932221 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:06.932223 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:06.932226 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:55:06.932309 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:06.932311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.932969 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.932970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:06.933893 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:06.933895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.934469 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.934470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:06.934545 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:06.934549 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 10:55:06.934604 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 10:55:06.934648 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":1,"thread_start_us":61,"threads_started":1}
I20260929 10:55:06.934787 19212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:06.931848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:35295 (local address 127.10.238.193:34399)
0929 10:55:06.931898 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:06.931900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:06.931909 (+ 9us) server_negotiation.cc:407] Connection header received
0929 10:55:06.931942 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:06.931943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:06.931951 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:06.932003 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:55:06.932237 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.932810 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.933838 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.934470 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.934544 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:06.934551 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:06.934684 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 10:55:06.934719 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260929 10:55:06.934752 19080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:06.931762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:34399 (local address 127.10.238.195:35295)
0929 10:55:06.931824 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:06.931837 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:06.931880 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:06.932055 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:06.932055 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:06.932058 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:55:06.932138 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:06.932140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.932893 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.932894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:06.933761 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:06.933763 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.934534 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.934535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:06.934612 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:06.934614 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 10:55:06.934665 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 10:55:06.934695 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":3}
I20260929 10:55:06.935101 18861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "333c3468bd974ece948b9f9dafafec15" candidate_uuid: "44a08b4b7d1c4435bee2560269289eb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62f127242e214ef497866289454389c2" is_pre_election: true
I20260929 10:55:06.935197 18861 raft_consensus.cc:2454] T 333c3468bd974ece948b9f9dafafec15 P 62f127242e214ef497866289454389c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44a08b4b7d1c4435bee2560269289eb3 in term 0.
I20260929 10:55:06.935227 19001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "333c3468bd974ece948b9f9dafafec15" candidate_uuid: "44a08b4b7d1c4435bee2560269289eb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a033d9467d4d451586b0506fff8ebdf9" is_pre_election: true
I20260929 10:55:06.935393 19001 raft_consensus.cc:2454] T 333c3468bd974ece948b9f9dafafec15 P a033d9467d4d451586b0506fff8ebdf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44a08b4b7d1c4435bee2560269289eb3 in term 0.
I20260929 10:55:06.935645 19073 leader_election.cc:302] T 333c3468bd974ece948b9f9dafafec15 P 44a08b4b7d1c4435bee2560269289eb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44a08b4b7d1c4435bee2560269289eb3, 62f127242e214ef497866289454389c2; no voters:
I20260929 10:55:06.935791 19220 raft_consensus.cc:2781] T 333c3468bd974ece948b9f9dafafec15 P 44a08b4b7d1c4435bee2560269289eb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:55:06.935848 19220 raft_consensus.cc:482] T 333c3468bd974ece948b9f9dafafec15 P 44a08b4b7d1c4435bee2560269289eb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:55:06.935864 19220 raft_consensus.cc:3037] T 333c3468bd974ece948b9f9dafafec15 P 44a08b4b7d1c4435bee2560269289eb3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:55:06.936517 19220 raft_consensus.cc:504] T 333c3468bd974ece948b9f9dafafec15 P 44a08b4b7d1c4435bee2560269289eb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } }
I20260929 10:55:06.936668 19220 leader_election.cc:288] T 333c3468bd974ece948b9f9dafafec15 P 44a08b4b7d1c4435bee2560269289eb3 [CANDIDATE]: Term 1 election: Requested vote from peers 62f127242e214ef497866289454389c2 (127.10.238.193:34399), a033d9467d4d451586b0506fff8ebdf9 (127.10.238.194:35881)
I20260929 10:55:06.936802 18861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "333c3468bd974ece948b9f9dafafec15" candidate_uuid: "44a08b4b7d1c4435bee2560269289eb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62f127242e214ef497866289454389c2"
I20260929 10:55:06.936818 19001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "333c3468bd974ece948b9f9dafafec15" candidate_uuid: "44a08b4b7d1c4435bee2560269289eb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a033d9467d4d451586b0506fff8ebdf9"
I20260929 10:55:06.936882 19001 raft_consensus.cc:3037] T 333c3468bd974ece948b9f9dafafec15 P a033d9467d4d451586b0506fff8ebdf9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:55:06.936882 18861 raft_consensus.cc:3037] T 333c3468bd974ece948b9f9dafafec15 P 62f127242e214ef497866289454389c2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:55:06.937656 18861 raft_consensus.cc:2454] T 333c3468bd974ece948b9f9dafafec15 P 62f127242e214ef497866289454389c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44a08b4b7d1c4435bee2560269289eb3 in term 1.
I20260929 10:55:06.937723 19001 raft_consensus.cc:2454] T 333c3468bd974ece948b9f9dafafec15 P a033d9467d4d451586b0506fff8ebdf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44a08b4b7d1c4435bee2560269289eb3 in term 1.
I20260929 10:55:06.937834 19073 leader_election.cc:302] T 333c3468bd974ece948b9f9dafafec15 P 44a08b4b7d1c4435bee2560269289eb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44a08b4b7d1c4435bee2560269289eb3, 62f127242e214ef497866289454389c2; no voters:
I20260929 10:55:06.937912 19220 raft_consensus.cc:2781] T 333c3468bd974ece948b9f9dafafec15 P 44a08b4b7d1c4435bee2560269289eb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:55:06.938020 19220 raft_consensus.cc:686] T 333c3468bd974ece948b9f9dafafec15 P 44a08b4b7d1c4435bee2560269289eb3 [term 1 LEADER]: Becoming Leader. State: Replica: 44a08b4b7d1c4435bee2560269289eb3, State: Running, Role: LEADER
I20260929 10:55:06.938074 19220 consensus_queue.cc:237] T 333c3468bd974ece948b9f9dafafec15 P 44a08b4b7d1c4435bee2560269289eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } }
I20260929 10:55:06.938836 18703 catalog_manager.cc:5495] T 333c3468bd974ece948b9f9dafafec15 P 44a08b4b7d1c4435bee2560269289eb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44a08b4b7d1c4435bee2560269289eb3 (127.10.238.195). New cstate: current_term: 1 leader_uuid: "44a08b4b7d1c4435bee2560269289eb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } health_report { overall_health: HEALTHY } } }
Sep 29 10:55:06 dist-test-slave-8nrt krb5kdc[18659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679306, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.238.254@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM
I20260929 10:55:06.993211 18753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:06.986057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33040 (local address 127.10.238.254:36571)
0929 10:55:06.986144 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:06.986146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:06.986265 (+ 119us) server_negotiation.cc:407] Connection header received
0929 10:55:06.986360 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:06.986361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:06.986371 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:06.986450 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:06.986931 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.987427 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.988141 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.988329 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.988418 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:06.990610 (+ 2192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:06.990611 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:06.990611 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:06.990617 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:06.992000 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:06.992412 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:06.992413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:06.992413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:06.992468 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:06.992660 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:06.992661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:06.992662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:06.992768 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:06.992843 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:06.992995 (+ 152us) server_negotiation.cc:299] Negotiation successful
0929 10:55:06.993078 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 10:55:06.993340 19251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:06.985964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:36571 (local address 127.0.0.1:33040)
0929 10:55:06.986184 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:06.986194 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:06.986286 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:06.986453 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:06.986454 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:06.986792 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:06.986842 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:06.986845 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.987521 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.987521 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:06.988047 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:06.988049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:06.988418 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:06.988419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:06.988483 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:06.989070 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:06.989074 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:06.990509 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:55:06.992102 (+ 1593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:06.992110 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:06.992113 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:06.992325 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:06.992533 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:06.992534 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:06.992534 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:06.992596 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:06.992835 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:55:06.992842 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:55:06.992886 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 10:55:06.992974 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 10:55:06.993154 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":79,"thread_start_us":92,"threads_started":1}
I20260929 10:55:06.995518 18703 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:33040:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 10:55:06.995749 18703 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:55:06.997061 18981 tablet_service.cc:1467] Processing CreateTablet for tablet 63e3cfbde41e4a25b5a13f088e035462 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=78ef07c79fd0418aac6fec7a266736c6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 10:55:06.997120 19121 tablet_service.cc:1467] Processing CreateTablet for tablet 63e3cfbde41e4a25b5a13f088e035462 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=78ef07c79fd0418aac6fec7a266736c6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 10:55:06.997241 18981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63e3cfbde41e4a25b5a13f088e035462. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:55:06.997292 19121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63e3cfbde41e4a25b5a13f088e035462. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:55:06.998379 19221 tablet_bootstrap.cc:492] T 63e3cfbde41e4a25b5a13f088e035462 P a033d9467d4d451586b0506fff8ebdf9: Bootstrap starting.
I20260929 10:55:06.998428 19215 tablet_bootstrap.cc:492] T 63e3cfbde41e4a25b5a13f088e035462 P 44a08b4b7d1c4435bee2560269289eb3: Bootstrap starting.
I20260929 10:55:06.998484 18841 tablet_service.cc:1467] Processing CreateTablet for tablet 63e3cfbde41e4a25b5a13f088e035462 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=78ef07c79fd0418aac6fec7a266736c6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 10:55:06.998618 18841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63e3cfbde41e4a25b5a13f088e035462. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:55:06.998747 19221 tablet_bootstrap.cc:654] T 63e3cfbde41e4a25b5a13f088e035462 P a033d9467d4d451586b0506fff8ebdf9: Neither blocks nor log segments found. Creating new log.
I20260929 10:55:06.998817 19215 tablet_bootstrap.cc:654] T 63e3cfbde41e4a25b5a13f088e035462 P 44a08b4b7d1c4435bee2560269289eb3: Neither blocks nor log segments found. Creating new log.
I20260929 10:55:06.999680 19215 tablet_bootstrap.cc:492] T 63e3cfbde41e4a25b5a13f088e035462 P 44a08b4b7d1c4435bee2560269289eb3: No bootstrap required, opened a new log
I20260929 10:55:06.999720 19215 ts_tablet_manager.cc:1397] T 63e3cfbde41e4a25b5a13f088e035462 P 44a08b4b7d1c4435bee2560269289eb3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 10:55:06.999876 19215 raft_consensus.cc:348] T 63e3cfbde41e4a25b5a13f088e035462 P 44a08b4b7d1c4435bee2560269289eb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } }
I20260929 10:55:06.999935 19215 raft_consensus.cc:374] T 63e3cfbde41e4a25b5a13f088e035462 P 44a08b4b7d1c4435bee2560269289eb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:55:06.999949 19215 raft_consensus.cc:729] T 63e3cfbde41e4a25b5a13f088e035462 P 44a08b4b7d1c4435bee2560269289eb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44a08b4b7d1c4435bee2560269289eb3, State: Initialized, Role: FOLLOWER
I20260929 10:55:07.000015 19215 consensus_queue.cc:260] T 63e3cfbde41e4a25b5a13f088e035462 P 44a08b4b7d1c4435bee2560269289eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } }
I20260929 10:55:07.000110 19215 ts_tablet_manager.cc:1428] T 63e3cfbde41e4a25b5a13f088e035462 P 44a08b4b7d1c4435bee2560269289eb3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:07.000149 19220 tablet_replica.cc:439] T 63e3cfbde41e4a25b5a13f088e035462 P 44a08b4b7d1c4435bee2560269289eb3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } } }
I20260929 10:55:07.000245 19220 tablet_replica.cc:442] T 63e3cfbde41e4a25b5a13f088e035462 P 44a08b4b7d1c4435bee2560269289eb3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:55:07.000463 19216 tablet_bootstrap.cc:492] T 63e3cfbde41e4a25b5a13f088e035462 P 62f127242e214ef497866289454389c2: Bootstrap starting.
I20260929 10:55:07.000612 19221 tablet_bootstrap.cc:492] T 63e3cfbde41e4a25b5a13f088e035462 P a033d9467d4d451586b0506fff8ebdf9: No bootstrap required, opened a new log
I20260929 10:55:07.000648 19221 ts_tablet_manager.cc:1397] T 63e3cfbde41e4a25b5a13f088e035462 P a033d9467d4d451586b0506fff8ebdf9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 10:55:07.000728 19221 raft_consensus.cc:348] T 63e3cfbde41e4a25b5a13f088e035462 P a033d9467d4d451586b0506fff8ebdf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } }
I20260929 10:55:07.000771 19221 raft_consensus.cc:374] T 63e3cfbde41e4a25b5a13f088e035462 P a033d9467d4d451586b0506fff8ebdf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:55:07.000784 19221 raft_consensus.cc:729] T 63e3cfbde41e4a25b5a13f088e035462 P a033d9467d4d451586b0506fff8ebdf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a033d9467d4d451586b0506fff8ebdf9, State: Initialized, Role: FOLLOWER
I20260929 10:55:07.000782 19216 tablet_bootstrap.cc:654] T 63e3cfbde41e4a25b5a13f088e035462 P 62f127242e214ef497866289454389c2: Neither blocks nor log segments found. Creating new log.
I20260929 10:55:07.000818 19221 consensus_queue.cc:260] T 63e3cfbde41e4a25b5a13f088e035462 P a033d9467d4d451586b0506fff8ebdf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } }
I20260929 10:55:07.000873 19221 ts_tablet_manager.cc:1428] T 63e3cfbde41e4a25b5a13f088e035462 P a033d9467d4d451586b0506fff8ebdf9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:07.000946 19225 tablet_replica.cc:439] T 63e3cfbde41e4a25b5a13f088e035462 P a033d9467d4d451586b0506fff8ebdf9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } } }
I20260929 10:55:07.001048 19225 tablet_replica.cc:442] T 63e3cfbde41e4a25b5a13f088e035462 P a033d9467d4d451586b0506fff8ebdf9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:55:07.001643 19216 tablet_bootstrap.cc:492] T 63e3cfbde41e4a25b5a13f088e035462 P 62f127242e214ef497866289454389c2: No bootstrap required, opened a new log
I20260929 10:55:07.001713 19216 ts_tablet_manager.cc:1397] T 63e3cfbde41e4a25b5a13f088e035462 P 62f127242e214ef497866289454389c2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 10:55:07.001859 19216 raft_consensus.cc:348] T 63e3cfbde41e4a25b5a13f088e035462 P 62f127242e214ef497866289454389c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } }
I20260929 10:55:07.001920 19216 raft_consensus.cc:374] T 63e3cfbde41e4a25b5a13f088e035462 P 62f127242e214ef497866289454389c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:55:07.001960 19216 raft_consensus.cc:729] T 63e3cfbde41e4a25b5a13f088e035462 P 62f127242e214ef497866289454389c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62f127242e214ef497866289454389c2, State: Initialized, Role: FOLLOWER
I20260929 10:55:07.002009 19216 consensus_queue.cc:260] T 63e3cfbde41e4a25b5a13f088e035462 P 62f127242e214ef497866289454389c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } }
I20260929 10:55:07.002128 19216 ts_tablet_manager.cc:1428] T 63e3cfbde41e4a25b5a13f088e035462 P 62f127242e214ef497866289454389c2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:07.002185 19219 tablet_replica.cc:439] T 63e3cfbde41e4a25b5a13f088e035462 P 62f127242e214ef497866289454389c2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } } }
I20260929 10:55:07.002260 19219 tablet_replica.cc:442] T 63e3cfbde41e4a25b5a13f088e035462 P 62f127242e214ef497866289454389c2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:55:07.144711 19225 raft_consensus.cc:482] T 63e3cfbde41e4a25b5a13f088e035462 P a033d9467d4d451586b0506fff8ebdf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:55:07.144817 19225 raft_consensus.cc:504] T 63e3cfbde41e4a25b5a13f088e035462 P a033d9467d4d451586b0506fff8ebdf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } }
I20260929 10:55:07.145009 19225 leader_election.cc:288] T 63e3cfbde41e4a25b5a13f088e035462 P a033d9467d4d451586b0506fff8ebdf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62f127242e214ef497866289454389c2 (127.10.238.193:34399), 44a08b4b7d1c4435bee2560269289eb3 (127.10.238.195:38935)
I20260929 10:55:07.145304 18861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63e3cfbde41e4a25b5a13f088e035462" candidate_uuid: "a033d9467d4d451586b0506fff8ebdf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62f127242e214ef497866289454389c2" is_pre_election: true
I20260929 10:55:07.145391 18861 raft_consensus.cc:2454] T 63e3cfbde41e4a25b5a13f088e035462 P 62f127242e214ef497866289454389c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a033d9467d4d451586b0506fff8ebdf9 in term 0.
I20260929 10:55:07.145314 19140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63e3cfbde41e4a25b5a13f088e035462" candidate_uuid: "a033d9467d4d451586b0506fff8ebdf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44a08b4b7d1c4435bee2560269289eb3" is_pre_election: true
I20260929 10:55:07.145443 19140 raft_consensus.cc:2454] T 63e3cfbde41e4a25b5a13f088e035462 P 44a08b4b7d1c4435bee2560269289eb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a033d9467d4d451586b0506fff8ebdf9 in term 0.
I20260929 10:55:07.145576 18933 leader_election.cc:302] T 63e3cfbde41e4a25b5a13f088e035462 P a033d9467d4d451586b0506fff8ebdf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62f127242e214ef497866289454389c2, a033d9467d4d451586b0506fff8ebdf9; no voters:
I20260929 10:55:07.145705 19225 raft_consensus.cc:2781] T 63e3cfbde41e4a25b5a13f088e035462 P a033d9467d4d451586b0506fff8ebdf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:55:07.145766 19225 raft_consensus.cc:482] T 63e3cfbde41e4a25b5a13f088e035462 P a033d9467d4d451586b0506fff8ebdf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:55:07.145780 19225 raft_consensus.cc:3037] T 63e3cfbde41e4a25b5a13f088e035462 P a033d9467d4d451586b0506fff8ebdf9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:55:07.146487 19225 raft_consensus.cc:504] T 63e3cfbde41e4a25b5a13f088e035462 P a033d9467d4d451586b0506fff8ebdf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } }
I20260929 10:55:07.146628 19225 leader_election.cc:288] T 63e3cfbde41e4a25b5a13f088e035462 P a033d9467d4d451586b0506fff8ebdf9 [CANDIDATE]: Term 1 election: Requested vote from peers 62f127242e214ef497866289454389c2 (127.10.238.193:34399), 44a08b4b7d1c4435bee2560269289eb3 (127.10.238.195:38935)
I20260929 10:55:07.146775 18861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63e3cfbde41e4a25b5a13f088e035462" candidate_uuid: "a033d9467d4d451586b0506fff8ebdf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62f127242e214ef497866289454389c2"
I20260929 10:55:07.146857 18861 raft_consensus.cc:3037] T 63e3cfbde41e4a25b5a13f088e035462 P 62f127242e214ef497866289454389c2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:55:07.146818 19140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63e3cfbde41e4a25b5a13f088e035462" candidate_uuid: "a033d9467d4d451586b0506fff8ebdf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44a08b4b7d1c4435bee2560269289eb3"
I20260929 10:55:07.146924 19140 raft_consensus.cc:3037] T 63e3cfbde41e4a25b5a13f088e035462 P 44a08b4b7d1c4435bee2560269289eb3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:55:07.147485 18861 raft_consensus.cc:2454] T 63e3cfbde41e4a25b5a13f088e035462 P 62f127242e214ef497866289454389c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a033d9467d4d451586b0506fff8ebdf9 in term 1.
I20260929 10:55:07.147660 18933 leader_election.cc:302] T 63e3cfbde41e4a25b5a13f088e035462 P a033d9467d4d451586b0506fff8ebdf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62f127242e214ef497866289454389c2, a033d9467d4d451586b0506fff8ebdf9; no voters:
I20260929 10:55:07.147660 19140 raft_consensus.cc:2454] T 63e3cfbde41e4a25b5a13f088e035462 P 44a08b4b7d1c4435bee2560269289eb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a033d9467d4d451586b0506fff8ebdf9 in term 1.
I20260929 10:55:07.147764 19225 raft_consensus.cc:2781] T 63e3cfbde41e4a25b5a13f088e035462 P a033d9467d4d451586b0506fff8ebdf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:55:07.147845 19225 raft_consensus.cc:686] T 63e3cfbde41e4a25b5a13f088e035462 P a033d9467d4d451586b0506fff8ebdf9 [term 1 LEADER]: Becoming Leader. State: Replica: a033d9467d4d451586b0506fff8ebdf9, State: Running, Role: LEADER
I20260929 10:55:07.147897 19225 consensus_queue.cc:237] T 63e3cfbde41e4a25b5a13f088e035462 P a033d9467d4d451586b0506fff8ebdf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } }
I20260929 10:55:07.148180 19223 tablet_replica.cc:439] T 63e3cfbde41e4a25b5a13f088e035462 P a033d9467d4d451586b0506fff8ebdf9: TxnStatusTablet state changed. Reason: New leader a033d9467d4d451586b0506fff8ebdf9. Latest consensus state: current_term: 1 leader_uuid: "a033d9467d4d451586b0506fff8ebdf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } } }
I20260929 10:55:07.148281 19223 tablet_replica.cc:442] T 63e3cfbde41e4a25b5a13f088e035462 P a033d9467d4d451586b0506fff8ebdf9: This TxnStatusTablet replica's current role is: LEADER
I20260929 10:55:07.148430 19256 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 10:55:07.148578 19256 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 10:55:07.148527 18704 catalog_manager.cc:5495] T 63e3cfbde41e4a25b5a13f088e035462 P a033d9467d4d451586b0506fff8ebdf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a033d9467d4d451586b0506fff8ebdf9 (127.10.238.194). New cstate: current_term: 1 leader_uuid: "a033d9467d4d451586b0506fff8ebdf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } health_report { overall_health: UNKNOWN } } }
I20260929 10:55:07.183990 19211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:07.181310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35314 (local address 127.10.238.194:35881)
0929 10:55:07.181362 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:07.181364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:07.181447 (+ 83us) server_negotiation.cc:407] Connection header received
0929 10:55:07.181546 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:07.181547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:07.181557 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:07.181603 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:55:07.181735 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:07.182206 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:07.183308 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:07.183499 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:07.183555 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:07.183760 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:55:07.183790 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:55:07.183827 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:07.183908 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 10:55:07.183925 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 10:55:07.184046 19251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:07.181205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.194:35881 (local address 127.0.0.1:35314)
0929 10:55:07.181392 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:07.181405 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:07.181469 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:07.181612 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:07.181613 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:07.181614 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 10:55:07.181672 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:07.181675 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:07.182283 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:07.182285 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:07.183212 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:07.183214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:07.183577 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:07.183577 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:07.183647 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:07.183672 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 10:55:07.183850 (+ 178us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 10:55:07.183858 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 10:55:07.183907 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 10:55:07.183955 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136}
I20260929 10:55:07.185134 18861 raft_consensus.cc:1261] T 63e3cfbde41e4a25b5a13f088e035462 P 62f127242e214ef497866289454389c2 [term 1 FOLLOWER]: Refusing update from remote peer a033d9467d4d451586b0506fff8ebdf9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:55:07.185190 19140 raft_consensus.cc:1261] T 63e3cfbde41e4a25b5a13f088e035462 P 44a08b4b7d1c4435bee2560269289eb3 [term 1 FOLLOWER]: Refusing update from remote peer a033d9467d4d451586b0506fff8ebdf9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:55:07.185483 19225 consensus_queue.cc:1035] T 63e3cfbde41e4a25b5a13f088e035462 P a033d9467d4d451586b0506fff8ebdf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:55:07.185611 19223 consensus_queue.cc:1035] T 63e3cfbde41e4a25b5a13f088e035462 P a033d9467d4d451586b0506fff8ebdf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:55:07.186168 19220 tablet_replica.cc:439] T 63e3cfbde41e4a25b5a13f088e035462 P 44a08b4b7d1c4435bee2560269289eb3: TxnStatusTablet state changed. Reason: New leader a033d9467d4d451586b0506fff8ebdf9. Latest consensus state: current_term: 1 leader_uuid: "a033d9467d4d451586b0506fff8ebdf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } } }
I20260929 10:55:07.186266 19220 tablet_replica.cc:442] T 63e3cfbde41e4a25b5a13f088e035462 P 44a08b4b7d1c4435bee2560269289eb3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:55:07.186239 19219 tablet_replica.cc:439] T 63e3cfbde41e4a25b5a13f088e035462 P 62f127242e214ef497866289454389c2: TxnStatusTablet state changed. Reason: New leader a033d9467d4d451586b0506fff8ebdf9. Latest consensus state: current_term: 1 leader_uuid: "a033d9467d4d451586b0506fff8ebdf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } } }
I20260929 10:55:07.186359 19219 tablet_replica.cc:442] T 63e3cfbde41e4a25b5a13f088e035462 P 62f127242e214ef497866289454389c2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:55:07.187060 19223 tablet_replica.cc:439] T 63e3cfbde41e4a25b5a13f088e035462 P a033d9467d4d451586b0506fff8ebdf9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a033d9467d4d451586b0506fff8ebdf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } } }
I20260929 10:55:07.187144 19223 tablet_replica.cc:442] T 63e3cfbde41e4a25b5a13f088e035462 P a033d9467d4d451586b0506fff8ebdf9: This TxnStatusTablet replica's current role is: LEADER
I20260929 10:55:07.187493 19220 tablet_replica.cc:439] T 63e3cfbde41e4a25b5a13f088e035462 P 44a08b4b7d1c4435bee2560269289eb3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a033d9467d4d451586b0506fff8ebdf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } } }
I20260929 10:55:07.187582 19220 tablet_replica.cc:442] T 63e3cfbde41e4a25b5a13f088e035462 P 44a08b4b7d1c4435bee2560269289eb3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:55:07.188246 19225 tablet_replica.cc:439] T 63e3cfbde41e4a25b5a13f088e035462 P a033d9467d4d451586b0506fff8ebdf9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a033d9467d4d451586b0506fff8ebdf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } } }
I20260929 10:55:07.188333 19225 tablet_replica.cc:442] T 63e3cfbde41e4a25b5a13f088e035462 P a033d9467d4d451586b0506fff8ebdf9: This TxnStatusTablet replica's current role is: LEADER
I20260929 10:55:07.188969 19219 tablet_replica.cc:439] T 63e3cfbde41e4a25b5a13f088e035462 P 62f127242e214ef497866289454389c2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a033d9467d4d451586b0506fff8ebdf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } } }
I20260929 10:55:07.189062 19219 tablet_replica.cc:442] T 63e3cfbde41e4a25b5a13f088e035462 P 62f127242e214ef497866289454389c2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:55:07.191601 19214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:07.188889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53796 (local address 127.10.238.195:38935)
0929 10:55:07.188928 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:07.188930 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:07.188939 (+ 9us) server_negotiation.cc:407] Connection header received
0929 10:55:07.189015 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:07.189016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:07.189037 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:07.189115 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:55:07.189365 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:07.189830 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:07.190883 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:07.191072 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:07.191124 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:07.191284 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:55:07.191379 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:55:07.191429 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:07.191495 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 10:55:07.191511 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 10:55:07.196869 19264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:07.194105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.194:35881 (local address 127.10.238.195:59047)
0929 10:55:07.194300 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:07.194310 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:07.194390 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:07.194599 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:07.194600 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:07.194601 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 10:55:07.194660 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:07.194661 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:07.195406 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:07.195407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:07.196028 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:07.196029 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:07.196424 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:07.196425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:07.196495 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:07.196517 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 10:55:07.196696 (+ 179us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 10:55:07.196698 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 10:55:07.196756 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 10:55:07.196797 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":85,"threads_started":1}
I20260929 10:55:07.196854 19211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:07.194170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:59047 (local address 127.10.238.194:35881)
0929 10:55:07.194220 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:07.194222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:07.194377 (+ 155us) server_negotiation.cc:407] Connection header received
0929 10:55:07.194508 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:07.194509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:07.194520 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:07.194595 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:55:07.194758 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:07.195325 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:07.196116 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:07.196338 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:07.196425 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:07.196613 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:55:07.196648 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:55:07.196696 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:07.196761 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 10:55:07.196780 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 10:55:07.199148 18861 raft_consensus.cc:1261] T 333c3468bd974ece948b9f9dafafec15 P 62f127242e214ef497866289454389c2 [term 1 FOLLOWER]: Refusing update from remote peer 44a08b4b7d1c4435bee2560269289eb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:55:07.199199 19001 raft_consensus.cc:1261] T 333c3468bd974ece948b9f9dafafec15 P a033d9467d4d451586b0506fff8ebdf9 [term 1 FOLLOWER]: Refusing update from remote peer 44a08b4b7d1c4435bee2560269289eb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:55:07.199417 19220 consensus_queue.cc:1035] T 333c3468bd974ece948b9f9dafafec15 P 44a08b4b7d1c4435bee2560269289eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:55:07.199572 19244 consensus_queue.cc:1035] T 333c3468bd974ece948b9f9dafafec15 P 44a08b4b7d1c4435bee2560269289eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:55:07.202176 19228 mvcc.cc:205] Tried to move back new op lower bound from 7334622442286002176 to 7334622441218854912. Current Snapshot: MvccSnapshot[applied={T|T < 7334622442286002176}]
I20260929 10:55:07.207773 19214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:07.205428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:34787 (local address 127.10.238.195:38935)
0929 10:55:07.205464 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:07.205466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:07.205474 (+ 8us) server_negotiation.cc:407] Connection header received
0929 10:55:07.205524 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:07.205525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:07.205534 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:07.205585 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:55:07.205669 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:07.206293 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:07.207017 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:07.207290 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:07.207357 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:07.207550 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:55:07.207584 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:55:07.207649 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:07.207689 (+ 40us) server_negotiation.cc:299] Negotiation successful
0929 10:55:07.207707 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 10:55:07.207813 19269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:07.205247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.195:38935 (local address 127.10.238.194:34787)
0929 10:55:07.205412 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:07.205423 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:07.205472 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:07.205590 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:07.205591 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:07.205592 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 10:55:07.205630 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:07.205631 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:07.206368 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:07.206369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:07.206925 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:07.206926 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:07.207389 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:07.207390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:07.207453 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:07.207469 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 10:55:07.207650 (+ 181us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 10:55:07.207651 (+ 1us) client_negotiation.cc:769] Sending connection context
0929 10:55:07.207678 (+ 27us) client_negotiation.cc:240] Negotiation successful
0929 10:55:07.207712 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":98,"threads_started":1}
I20260929 10:55:07.217351 18703 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:55:07.217459 18703 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:55:07.218089 18703 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f5072e103f2c4e39a27e638aa276acb7: Sending DeleteTablet for 3 replicas of tablet 333c3468bd974ece948b9f9dafafec15
I20260929 10:55:07.218343 19121 tablet_service.cc:1514] Processing DeleteTablet for tablet 333c3468bd974ece948b9f9dafafec15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:55:07 UTC) from {username='oryx', principal='oryx/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:53794
I20260929 10:55:07.218343 18841 tablet_service.cc:1514] Processing DeleteTablet for tablet 333c3468bd974ece948b9f9dafafec15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:55:07 UTC) from {username='oryx', principal='oryx/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:59466
I20260929 10:55:07.218430 19239 tablet_replica.cc:330] T 333c3468bd974ece948b9f9dafafec15 P 62f127242e214ef497866289454389c2: stopping tablet replica
I20260929 10:55:07.218441 18981 tablet_service.cc:1514] Processing DeleteTablet for tablet 333c3468bd974ece948b9f9dafafec15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:55:07 UTC) from {username='oryx', principal='oryx/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:35296
I20260929 10:55:07.218461 19238 tablet_replica.cc:330] T 333c3468bd974ece948b9f9dafafec15 P 44a08b4b7d1c4435bee2560269289eb3: stopping tablet replica
I20260929 10:55:07.218484 19239 raft_consensus.cc:2229] T 333c3468bd974ece948b9f9dafafec15 P 62f127242e214ef497866289454389c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:55:07.218524 19239 raft_consensus.cc:2258] T 333c3468bd974ece948b9f9dafafec15 P 62f127242e214ef497866289454389c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:55:07.218523 19238 raft_consensus.cc:2229] T 333c3468bd974ece948b9f9dafafec15 P 44a08b4b7d1c4435bee2560269289eb3 [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:55:07.218585 19237 tablet_replica.cc:330] T 333c3468bd974ece948b9f9dafafec15 P a033d9467d4d451586b0506fff8ebdf9: stopping tablet replica
I20260929 10:55:07.218604 19238 raft_consensus.cc:2258] T 333c3468bd974ece948b9f9dafafec15 P 44a08b4b7d1c4435bee2560269289eb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:55:07.218631 19237 raft_consensus.cc:2229] T 333c3468bd974ece948b9f9dafafec15 P a033d9467d4d451586b0506fff8ebdf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:55:07.218663 19237 raft_consensus.cc:2258] T 333c3468bd974ece948b9f9dafafec15 P a033d9467d4d451586b0506fff8ebdf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:55:07.219059 19238 ts_tablet_manager.cc:1905] T 333c3468bd974ece948b9f9dafafec15 P 44a08b4b7d1c4435bee2560269289eb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:55:07.219449 19237 ts_tablet_manager.cc:1905] T 333c3468bd974ece948b9f9dafafec15 P a033d9467d4d451586b0506fff8ebdf9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:55:07.220698 19238 ts_tablet_manager.cc:1918] T 333c3468bd974ece948b9f9dafafec15 P 44a08b4b7d1c4435bee2560269289eb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 10:55:07.220750 19238 log.cc:1196] T 333c3468bd974ece948b9f9dafafec15 P 44a08b4b7d1c4435bee2560269289eb3: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-2/wal/wals/333c3468bd974ece948b9f9dafafec15
I20260929 10:55:07.220990 19238 ts_tablet_manager.cc:1939] T 333c3468bd974ece948b9f9dafafec15 P 44a08b4b7d1c4435bee2560269289eb3: Deleting consensus metadata
I20260929 10:55:07.221006 19237 ts_tablet_manager.cc:1918] T 333c3468bd974ece948b9f9dafafec15 P a033d9467d4d451586b0506fff8ebdf9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 10:55:07.221060 19237 log.cc:1196] T 333c3468bd974ece948b9f9dafafec15 P a033d9467d4d451586b0506fff8ebdf9: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-1/wal/wals/333c3468bd974ece948b9f9dafafec15
I20260929 10:55:07.221314 19237 ts_tablet_manager.cc:1939] T 333c3468bd974ece948b9f9dafafec15 P a033d9467d4d451586b0506fff8ebdf9: Deleting consensus metadata
I20260929 10:55:07.221318 18690 catalog_manager.cc:4841] TS 44a08b4b7d1c4435bee2560269289eb3 (127.10.238.195:38935): tablet 333c3468bd974ece948b9f9dafafec15 (table test-table [id=48e4772c72614fd9937748b936efa2ec]) successfully deleted
I20260929 10:55:07.221649 18690 catalog_manager.cc:4841] TS a033d9467d4d451586b0506fff8ebdf9 (127.10.238.194:35881): tablet 333c3468bd974ece948b9f9dafafec15 (table test-table [id=48e4772c72614fd9937748b936efa2ec]) successfully deleted
I20260929 10:55:07.221724 19239 ts_tablet_manager.cc:1905] T 333c3468bd974ece948b9f9dafafec15 P 62f127242e214ef497866289454389c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:55:07.223335 19239 ts_tablet_manager.cc:1918] T 333c3468bd974ece948b9f9dafafec15 P 62f127242e214ef497866289454389c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 10:55:07.223384 19239 log.cc:1196] T 333c3468bd974ece948b9f9dafafec15 P 62f127242e214ef497866289454389c2: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-0/wal/wals/333c3468bd974ece948b9f9dafafec15
I20260929 10:55:07.223610 19239 ts_tablet_manager.cc:1939] T 333c3468bd974ece948b9f9dafafec15 P 62f127242e214ef497866289454389c2: Deleting consensus metadata
I20260929 10:55:07.223930 18689 catalog_manager.cc:4841] TS 62f127242e214ef497866289454389c2 (127.10.238.193:34399): tablet 333c3468bd974ece948b9f9dafafec15 (table test-table [id=48e4772c72614fd9937748b936efa2ec]) successfully deleted
Sep 29 10:55:07 dist-test-slave-8nrt krb5kdc[18659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 10:55:07 dist-test-slave-8nrt krb5kdc[18659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:07 dist-test-slave-8nrt krb5kdc[18659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
W20260929 10:55:07.255627 18753 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:07.248579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33046 (local address 127.10.238.254:36571)
0929 10:55:07.248630 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:07.248632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:07.248808 (+ 176us) server_negotiation.cc:407] Connection header received
0929 10:55:07.248874 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:07.248875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:07.248887 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:07.248962 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:07.249703 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:07.250500 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:07.251571 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:07.251872 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:07.251963 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:07.255555 (+ 3592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33046: BlockingRecv error: recv got EOF from 127.0.0.1:33046 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 29 10:55:07 dist-test-slave-8nrt krb5kdc[18659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM
I20260929 10:55:07.265772 18753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:07.256705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33058 (local address 127.10.238.254:36571)
0929 10:55:07.256745 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:07.256746 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:07.257011 (+ 265us) server_negotiation.cc:407] Connection header received
0929 10:55:07.257058 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:07.257059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:07.257070 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:07.257123 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:07.257747 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:07.258514 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:07.259516 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:07.259816 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:07.259887 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:07.262635 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:07.262636 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:07.262637 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:07.262644 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:07.264437 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:07.264838 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:07.264839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:07.264840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:07.264902 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:07.265120 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:07.265121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:07.265121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:07.265358 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:07.265462 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:07.265596 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 10:55:07.265675 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 10:55:07.267745 18703 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33058:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:55:07.267913 18703 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:55:07.269487 18981 tablet_service.cc:1467] Processing CreateTablet for tablet 1bc2efd01e814b819ec67cce613d5d4c (DEFAULT_TABLE table=test-table [id=5031ca5c68dc4d099842d19b182d2a9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:55:07.269629 18981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bc2efd01e814b819ec67cce613d5d4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:55:07.271076 18841 tablet_service.cc:1467] Processing CreateTablet for tablet 1bc2efd01e814b819ec67cce613d5d4c (DEFAULT_TABLE table=test-table [id=5031ca5c68dc4d099842d19b182d2a9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:55:07.271200 18841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bc2efd01e814b819ec67cce613d5d4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:55:07.272449 19121 tablet_service.cc:1467] Processing CreateTablet for tablet 1bc2efd01e814b819ec67cce613d5d4c (DEFAULT_TABLE table=test-table [id=5031ca5c68dc4d099842d19b182d2a9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:55:07.272562 19121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bc2efd01e814b819ec67cce613d5d4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:55:07.273348 19221 tablet_bootstrap.cc:492] T 1bc2efd01e814b819ec67cce613d5d4c P a033d9467d4d451586b0506fff8ebdf9: Bootstrap starting.
I20260929 10:55:07.273646 19221 tablet_bootstrap.cc:654] T 1bc2efd01e814b819ec67cce613d5d4c P a033d9467d4d451586b0506fff8ebdf9: Neither blocks nor log segments found. Creating new log.
I20260929 10:55:07.273902 19216 tablet_bootstrap.cc:492] T 1bc2efd01e814b819ec67cce613d5d4c P 62f127242e214ef497866289454389c2: Bootstrap starting.
I20260929 10:55:07.274312 19216 tablet_bootstrap.cc:654] T 1bc2efd01e814b819ec67cce613d5d4c P 62f127242e214ef497866289454389c2: Neither blocks nor log segments found. Creating new log.
I20260929 10:55:07.275132 19221 tablet_bootstrap.cc:492] T 1bc2efd01e814b819ec67cce613d5d4c P a033d9467d4d451586b0506fff8ebdf9: No bootstrap required, opened a new log
I20260929 10:55:07.275195 19221 ts_tablet_manager.cc:1397] T 1bc2efd01e814b819ec67cce613d5d4c P a033d9467d4d451586b0506fff8ebdf9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 10:55:07.275307 19221 raft_consensus.cc:348] T 1bc2efd01e814b819ec67cce613d5d4c P a033d9467d4d451586b0506fff8ebdf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } }
I20260929 10:55:07.275374 19221 raft_consensus.cc:374] T 1bc2efd01e814b819ec67cce613d5d4c P a033d9467d4d451586b0506fff8ebdf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:55:07.275393 19221 raft_consensus.cc:729] T 1bc2efd01e814b819ec67cce613d5d4c P a033d9467d4d451586b0506fff8ebdf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a033d9467d4d451586b0506fff8ebdf9, State: Initialized, Role: FOLLOWER
I20260929 10:55:07.275441 19221 consensus_queue.cc:260] T 1bc2efd01e814b819ec67cce613d5d4c P a033d9467d4d451586b0506fff8ebdf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } }
I20260929 10:55:07.275527 19221 ts_tablet_manager.cc:1428] T 1bc2efd01e814b819ec67cce613d5d4c P a033d9467d4d451586b0506fff8ebdf9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:07.276273 19216 tablet_bootstrap.cc:492] T 1bc2efd01e814b819ec67cce613d5d4c P 62f127242e214ef497866289454389c2: No bootstrap required, opened a new log
I20260929 10:55:07.276324 19216 ts_tablet_manager.cc:1397] T 1bc2efd01e814b819ec67cce613d5d4c P 62f127242e214ef497866289454389c2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 10:55:07.276489 19216 raft_consensus.cc:348] T 1bc2efd01e814b819ec67cce613d5d4c P 62f127242e214ef497866289454389c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } }
I20260929 10:55:07.276549 19216 raft_consensus.cc:374] T 1bc2efd01e814b819ec67cce613d5d4c P 62f127242e214ef497866289454389c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:55:07.276569 19216 raft_consensus.cc:729] T 1bc2efd01e814b819ec67cce613d5d4c P 62f127242e214ef497866289454389c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62f127242e214ef497866289454389c2, State: Initialized, Role: FOLLOWER
I20260929 10:55:07.276608 19216 consensus_queue.cc:260] T 1bc2efd01e814b819ec67cce613d5d4c P 62f127242e214ef497866289454389c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } }
I20260929 10:55:07.276690 19216 ts_tablet_manager.cc:1428] T 1bc2efd01e814b819ec67cce613d5d4c P 62f127242e214ef497866289454389c2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:07.278024 19215 tablet_bootstrap.cc:492] T 1bc2efd01e814b819ec67cce613d5d4c P 44a08b4b7d1c4435bee2560269289eb3: Bootstrap starting.
I20260929 10:55:07.278426 19215 tablet_bootstrap.cc:654] T 1bc2efd01e814b819ec67cce613d5d4c P 44a08b4b7d1c4435bee2560269289eb3: Neither blocks nor log segments found. Creating new log.
I20260929 10:55:07.279178 19215 tablet_bootstrap.cc:492] T 1bc2efd01e814b819ec67cce613d5d4c P 44a08b4b7d1c4435bee2560269289eb3: No bootstrap required, opened a new log
I20260929 10:55:07.279228 19215 ts_tablet_manager.cc:1397] T 1bc2efd01e814b819ec67cce613d5d4c P 44a08b4b7d1c4435bee2560269289eb3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 10:55:07.279390 19215 raft_consensus.cc:348] T 1bc2efd01e814b819ec67cce613d5d4c P 44a08b4b7d1c4435bee2560269289eb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } }
I20260929 10:55:07.279452 19215 raft_consensus.cc:374] T 1bc2efd01e814b819ec67cce613d5d4c P 44a08b4b7d1c4435bee2560269289eb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:55:07.279472 19215 raft_consensus.cc:729] T 1bc2efd01e814b819ec67cce613d5d4c P 44a08b4b7d1c4435bee2560269289eb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44a08b4b7d1c4435bee2560269289eb3, State: Initialized, Role: FOLLOWER
I20260929 10:55:07.279510 19215 consensus_queue.cc:260] T 1bc2efd01e814b819ec67cce613d5d4c P 44a08b4b7d1c4435bee2560269289eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } }
I20260929 10:55:07.279589 19215 ts_tablet_manager.cc:1428] T 1bc2efd01e814b819ec67cce613d5d4c P 44a08b4b7d1c4435bee2560269289eb3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:07.321084 19219 raft_consensus.cc:482] T 1bc2efd01e814b819ec67cce613d5d4c P 62f127242e214ef497866289454389c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:55:07.321182 19219 raft_consensus.cc:504] T 1bc2efd01e814b819ec67cce613d5d4c P 62f127242e214ef497866289454389c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } }
I20260929 10:55:07.321559 19219 leader_election.cc:288] T 1bc2efd01e814b819ec67cce613d5d4c P 62f127242e214ef497866289454389c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a033d9467d4d451586b0506fff8ebdf9 (127.10.238.194:35881), 44a08b4b7d1c4435bee2560269289eb3 (127.10.238.195:38935)
I20260929 10:55:07.325191 19214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:07.322083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:40683 (local address 127.10.238.195:38935)
0929 10:55:07.322129 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:07.322131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:07.322197 (+ 66us) server_negotiation.cc:407] Connection header received
0929 10:55:07.322260 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:07.322261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:07.322269 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:07.322328 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:55:07.322505 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:07.323225 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:07.324330 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:07.324828 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:07.324903 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:07.324912 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:07.325084 (+ 172us) server_negotiation.cc:299] Negotiation successful
0929 10:55:07.325120 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 10:55:07.325204 19288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:07.321995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.195:38935 (local address 127.10.238.193:40683)
0929 10:55:07.322110 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:07.322121 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:07.322196 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:07.322352 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:07.322353 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:07.322356 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:55:07.322422 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:07.322424 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:07.323312 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:07.323313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:07.324233 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:07.324236 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:07.324924 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:07.324925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:07.324990 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:07.324995 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 10:55:07.325059 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 10:55:07.325099 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":23,"thread_start_us":46,"threads_started":1}
I20260929 10:55:07.325634 19287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:07.321783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.194:35881 (local address 127.10.238.193:55401)
0929 10:55:07.321943 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:07.321958 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:07.322060 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:07.322268 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:07.322269 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:07.322272 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:55:07.322336 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:07.322339 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:07.323376 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:07.323378 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:07.324600 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:07.324604 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:07.325373 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:07.325374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:07.325452 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:07.325456 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 10:55:07.325509 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 10:55:07.325543 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":85,"threads_started":1}
I20260929 10:55:07.325654 19211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:07.321919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:55401 (local address 127.10.238.194:35881)
0929 10:55:07.321965 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:07.321967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:07.322027 (+ 60us) server_negotiation.cc:407] Connection header received
0929 10:55:07.322146 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:07.322147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:07.322157 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:07.322193 (+ 36us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:55:07.322395 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:07.323213 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:07.324693 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:07.325232 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:07.325324 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:07.325335 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:07.325542 (+ 207us) server_negotiation.cc:299] Negotiation successful
0929 10:55:07.325589 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 10:55:07.325724 19140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bc2efd01e814b819ec67cce613d5d4c" candidate_uuid: "62f127242e214ef497866289454389c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44a08b4b7d1c4435bee2560269289eb3" is_pre_election: true
I20260929 10:55:07.325821 19140 raft_consensus.cc:2454] T 1bc2efd01e814b819ec67cce613d5d4c P 44a08b4b7d1c4435bee2560269289eb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62f127242e214ef497866289454389c2 in term 0.
I20260929 10:55:07.325906 19001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bc2efd01e814b819ec67cce613d5d4c" candidate_uuid: "62f127242e214ef497866289454389c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a033d9467d4d451586b0506fff8ebdf9" is_pre_election: true
I20260929 10:55:07.325990 19001 raft_consensus.cc:2454] T 1bc2efd01e814b819ec67cce613d5d4c P a033d9467d4d451586b0506fff8ebdf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62f127242e214ef497866289454389c2 in term 0.
I20260929 10:55:07.325974 18794 leader_election.cc:302] T 1bc2efd01e814b819ec67cce613d5d4c P 62f127242e214ef497866289454389c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44a08b4b7d1c4435bee2560269289eb3, 62f127242e214ef497866289454389c2; no voters:
I20260929 10:55:07.326153 19219 raft_consensus.cc:2781] T 1bc2efd01e814b819ec67cce613d5d4c P 62f127242e214ef497866289454389c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:55:07.326193 19219 raft_consensus.cc:482] T 1bc2efd01e814b819ec67cce613d5d4c P 62f127242e214ef497866289454389c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:55:07.326212 19219 raft_consensus.cc:3037] T 1bc2efd01e814b819ec67cce613d5d4c P 62f127242e214ef497866289454389c2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:55:07.326942 19219 raft_consensus.cc:504] T 1bc2efd01e814b819ec67cce613d5d4c P 62f127242e214ef497866289454389c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } }
I20260929 10:55:07.327071 19219 leader_election.cc:288] T 1bc2efd01e814b819ec67cce613d5d4c P 62f127242e214ef497866289454389c2 [CANDIDATE]: Term 1 election: Requested vote from peers a033d9467d4d451586b0506fff8ebdf9 (127.10.238.194:35881), 44a08b4b7d1c4435bee2560269289eb3 (127.10.238.195:38935)
I20260929 10:55:07.327214 19001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bc2efd01e814b819ec67cce613d5d4c" candidate_uuid: "62f127242e214ef497866289454389c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a033d9467d4d451586b0506fff8ebdf9"
I20260929 10:55:07.327270 19140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bc2efd01e814b819ec67cce613d5d4c" candidate_uuid: "62f127242e214ef497866289454389c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44a08b4b7d1c4435bee2560269289eb3"
I20260929 10:55:07.327312 19001 raft_consensus.cc:3037] T 1bc2efd01e814b819ec67cce613d5d4c P a033d9467d4d451586b0506fff8ebdf9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:55:07.327337 19140 raft_consensus.cc:3037] T 1bc2efd01e814b819ec67cce613d5d4c P 44a08b4b7d1c4435bee2560269289eb3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:55:07.328064 19001 raft_consensus.cc:2454] T 1bc2efd01e814b819ec67cce613d5d4c P a033d9467d4d451586b0506fff8ebdf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62f127242e214ef497866289454389c2 in term 1.
I20260929 10:55:07.328064 19140 raft_consensus.cc:2454] T 1bc2efd01e814b819ec67cce613d5d4c P 44a08b4b7d1c4435bee2560269289eb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62f127242e214ef497866289454389c2 in term 1.
I20260929 10:55:07.328220 18794 leader_election.cc:302] T 1bc2efd01e814b819ec67cce613d5d4c P 62f127242e214ef497866289454389c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62f127242e214ef497866289454389c2, a033d9467d4d451586b0506fff8ebdf9; no voters:
I20260929 10:55:07.328359 19219 raft_consensus.cc:2781] T 1bc2efd01e814b819ec67cce613d5d4c P 62f127242e214ef497866289454389c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:55:07.328531 19219 raft_consensus.cc:686] T 1bc2efd01e814b819ec67cce613d5d4c P 62f127242e214ef497866289454389c2 [term 1 LEADER]: Becoming Leader. State: Replica: 62f127242e214ef497866289454389c2, State: Running, Role: LEADER
I20260929 10:55:07.328603 19219 consensus_queue.cc:237] T 1bc2efd01e814b819ec67cce613d5d4c P 62f127242e214ef497866289454389c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } }
I20260929 10:55:07.329330 18704 catalog_manager.cc:5495] T 1bc2efd01e814b819ec67cce613d5d4c P 62f127242e214ef497866289454389c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 62f127242e214ef497866289454389c2 (127.10.238.193). New cstate: current_term: 1 leader_uuid: "62f127242e214ef497866289454389c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } health_report { overall_health: UNKNOWN } } }
I20260929 10:55:07.339242 19212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:07.336584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59476 (local address 127.10.238.193:34399)
0929 10:55:07.336623 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:07.336625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:07.336633 (+ 8us) server_negotiation.cc:407] Connection header received
0929 10:55:07.336718 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:07.336719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:07.336738 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:07.336801 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:55:07.336929 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:07.337449 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:07.338570 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:07.338742 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:07.338798 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:07.338952 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:55:07.339045 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:55:07.339098 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:07.339171 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 10:55:07.339188 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 10:55:07.340232 19001 raft_consensus.cc:1261] T 1bc2efd01e814b819ec67cce613d5d4c P a033d9467d4d451586b0506fff8ebdf9 [term 1 FOLLOWER]: Refusing update from remote peer 62f127242e214ef497866289454389c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:55:07.340263 19140 raft_consensus.cc:1261] T 1bc2efd01e814b819ec67cce613d5d4c P 44a08b4b7d1c4435bee2560269289eb3 [term 1 FOLLOWER]: Refusing update from remote peer 62f127242e214ef497866289454389c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:55:07.340516 19289 consensus_queue.cc:1035] T 1bc2efd01e814b819ec67cce613d5d4c P 62f127242e214ef497866289454389c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:55:07.340663 19219 consensus_queue.cc:1035] T 1bc2efd01e814b819ec67cce613d5d4c P 62f127242e214ef497866289454389c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:55:07.344209 18704 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:55:07.344303 18704 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:55:07.345301 18704 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f5072e103f2c4e39a27e638aa276acb7: Sending DeleteTablet for 3 replicas of tablet 1bc2efd01e814b819ec67cce613d5d4c
I20260929 10:55:07.345544 18841 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bc2efd01e814b819ec67cce613d5d4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:55:07 UTC) from {username='oryx', principal='oryx/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:59466
I20260929 10:55:07.345630 19239 tablet_replica.cc:330] T 1bc2efd01e814b819ec67cce613d5d4c P 62f127242e214ef497866289454389c2: stopping tablet replica
I20260929 10:55:07.345674 19239 raft_consensus.cc:2229] T 1bc2efd01e814b819ec67cce613d5d4c P 62f127242e214ef497866289454389c2 [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:55:07.345754 19239 raft_consensus.cc:2258] T 1bc2efd01e814b819ec67cce613d5d4c P 62f127242e214ef497866289454389c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:55:07.345732 19120 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bc2efd01e814b819ec67cce613d5d4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:55:07 UTC) from {username='oryx', principal='oryx/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:53794
I20260929 10:55:07.345816 19238 tablet_replica.cc:330] T 1bc2efd01e814b819ec67cce613d5d4c P 44a08b4b7d1c4435bee2560269289eb3: stopping tablet replica
I20260929 10:55:07.345886 19238 raft_consensus.cc:2229] T 1bc2efd01e814b819ec67cce613d5d4c P 44a08b4b7d1c4435bee2560269289eb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:55:07.345825 19121 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bc2efd01e814b819ec67cce613d5d4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:55:07 UTC) from {username='oryx', principal='oryx/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:53794
I20260929 10:55:07.345935 19238 raft_consensus.cc:2258] T 1bc2efd01e814b819ec67cce613d5d4c P 44a08b4b7d1c4435bee2560269289eb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:55:07.346038 18981 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bc2efd01e814b819ec67cce613d5d4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:55:07 UTC) from {username='oryx', principal='oryx/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:35296
I20260929 10:55:07.346242 19237 tablet_replica.cc:330] T 1bc2efd01e814b819ec67cce613d5d4c P a033d9467d4d451586b0506fff8ebdf9: stopping tablet replica
I20260929 10:55:07.346305 19237 raft_consensus.cc:2229] T 1bc2efd01e814b819ec67cce613d5d4c P a033d9467d4d451586b0506fff8ebdf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:55:07.346347 19237 raft_consensus.cc:2258] T 1bc2efd01e814b819ec67cce613d5d4c P a033d9467d4d451586b0506fff8ebdf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:55:07.346335 19238 ts_tablet_manager.cc:1905] T 1bc2efd01e814b819ec67cce613d5d4c P 44a08b4b7d1c4435bee2560269289eb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:55:07.346675 19237 ts_tablet_manager.cc:1905] T 1bc2efd01e814b819ec67cce613d5d4c P a033d9467d4d451586b0506fff8ebdf9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:55:07.346711 19239 ts_tablet_manager.cc:1905] T 1bc2efd01e814b819ec67cce613d5d4c P 62f127242e214ef497866289454389c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:55:07.347918 19238 ts_tablet_manager.cc:1918] T 1bc2efd01e814b819ec67cce613d5d4c P 44a08b4b7d1c4435bee2560269289eb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:55:07.347975 19238 log.cc:1196] T 1bc2efd01e814b819ec67cce613d5d4c P 44a08b4b7d1c4435bee2560269289eb3: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-2/wal/wals/1bc2efd01e814b819ec67cce613d5d4c
I20260929 10:55:07.348205 19238 ts_tablet_manager.cc:1939] T 1bc2efd01e814b819ec67cce613d5d4c P 44a08b4b7d1c4435bee2560269289eb3: Deleting consensus metadata
I20260929 10:55:07.348219 19237 ts_tablet_manager.cc:1918] T 1bc2efd01e814b819ec67cce613d5d4c P a033d9467d4d451586b0506fff8ebdf9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:55:07.348232 19239 ts_tablet_manager.cc:1918] T 1bc2efd01e814b819ec67cce613d5d4c P 62f127242e214ef497866289454389c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:55:07.348287 19239 log.cc:1196] T 1bc2efd01e814b819ec67cce613d5d4c P 62f127242e214ef497866289454389c2: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-0/wal/wals/1bc2efd01e814b819ec67cce613d5d4c
I20260929 10:55:07.348284 19237 log.cc:1196] T 1bc2efd01e814b819ec67cce613d5d4c P a033d9467d4d451586b0506fff8ebdf9: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-1/wal/wals/1bc2efd01e814b819ec67cce613d5d4c
I20260929 10:55:07.348508 19239 ts_tablet_manager.cc:1939] T 1bc2efd01e814b819ec67cce613d5d4c P 62f127242e214ef497866289454389c2: Deleting consensus metadata
I20260929 10:55:07.348513 19237 ts_tablet_manager.cc:1939] T 1bc2efd01e814b819ec67cce613d5d4c P a033d9467d4d451586b0506fff8ebdf9: Deleting consensus metadata
I20260929 10:55:07.348809 18689 catalog_manager.cc:4841] TS 62f127242e214ef497866289454389c2 (127.10.238.193:34399): tablet 1bc2efd01e814b819ec67cce613d5d4c (table test-table [id=5031ca5c68dc4d099842d19b182d2a9d]) successfully deleted
I20260929 10:55:07.348970 18690 catalog_manager.cc:4841] TS a033d9467d4d451586b0506fff8ebdf9 (127.10.238.194:35881): tablet 1bc2efd01e814b819ec67cce613d5d4c (table test-table [id=5031ca5c68dc4d099842d19b182d2a9d]) successfully deleted
I20260929 10:55:07.349050 18690 catalog_manager.cc:4841] TS 44a08b4b7d1c4435bee2560269289eb3 (127.10.238.195:38935): tablet 1bc2efd01e814b819ec67cce613d5d4c (table test-table [id=5031ca5c68dc4d099842d19b182d2a9d]) successfully deleted
W20260929 10:55:07.349140 18690 catalog_manager.cc:4808] TS 44a08b4b7d1c4435bee2560269289eb3 (127.10.238.195:38935): delete failed for tablet 1bc2efd01e814b819ec67cce613d5d4c because the tablet was not found. No further retry: Not found: Tablet not found: 1bc2efd01e814b819ec67cce613d5d4c
I20260929 10:55:07.349308 18704 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33058:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:55:07.349466 18704 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:55:07.350673 18841 tablet_service.cc:1467] Processing CreateTablet for tablet 7056a1619d5a4083a8215e8d3a8165c2 (DEFAULT_TABLE table=test-table [id=9aa903b2d47f4d33bb0aae38dadbfde1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:55:07.350787 18841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7056a1619d5a4083a8215e8d3a8165c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:55:07.352007 18981 tablet_service.cc:1467] Processing CreateTablet for tablet 7056a1619d5a4083a8215e8d3a8165c2 (DEFAULT_TABLE table=test-table [id=9aa903b2d47f4d33bb0aae38dadbfde1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:55:07.352103 18981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7056a1619d5a4083a8215e8d3a8165c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:55:07.353412 19121 tablet_service.cc:1467] Processing CreateTablet for tablet 7056a1619d5a4083a8215e8d3a8165c2 (DEFAULT_TABLE table=test-table [id=9aa903b2d47f4d33bb0aae38dadbfde1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:55:07.353510 19121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7056a1619d5a4083a8215e8d3a8165c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:55:07.354729 19216 tablet_bootstrap.cc:492] T 7056a1619d5a4083a8215e8d3a8165c2 P 62f127242e214ef497866289454389c2: Bootstrap starting.
I20260929 10:55:07.355129 19216 tablet_bootstrap.cc:654] T 7056a1619d5a4083a8215e8d3a8165c2 P 62f127242e214ef497866289454389c2: Neither blocks nor log segments found. Creating new log.
I20260929 10:55:07.355441 19221 tablet_bootstrap.cc:492] T 7056a1619d5a4083a8215e8d3a8165c2 P a033d9467d4d451586b0506fff8ebdf9: Bootstrap starting.
I20260929 10:55:07.355713 19221 tablet_bootstrap.cc:654] T 7056a1619d5a4083a8215e8d3a8165c2 P a033d9467d4d451586b0506fff8ebdf9: Neither blocks nor log segments found. Creating new log.
I20260929 10:55:07.355917 19215 tablet_bootstrap.cc:492] T 7056a1619d5a4083a8215e8d3a8165c2 P 44a08b4b7d1c4435bee2560269289eb3: Bootstrap starting.
I20260929 10:55:07.356310 19215 tablet_bootstrap.cc:654] T 7056a1619d5a4083a8215e8d3a8165c2 P 44a08b4b7d1c4435bee2560269289eb3: Neither blocks nor log segments found. Creating new log.
I20260929 10:55:07.356799 19221 tablet_bootstrap.cc:492] T 7056a1619d5a4083a8215e8d3a8165c2 P a033d9467d4d451586b0506fff8ebdf9: No bootstrap required, opened a new log
I20260929 10:55:07.356842 19221 ts_tablet_manager.cc:1397] T 7056a1619d5a4083a8215e8d3a8165c2 P a033d9467d4d451586b0506fff8ebdf9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 10:55:07.356938 19221 raft_consensus.cc:348] T 7056a1619d5a4083a8215e8d3a8165c2 P a033d9467d4d451586b0506fff8ebdf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } }
I20260929 10:55:07.356999 19221 raft_consensus.cc:374] T 7056a1619d5a4083a8215e8d3a8165c2 P a033d9467d4d451586b0506fff8ebdf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:55:07.357017 19221 raft_consensus.cc:729] T 7056a1619d5a4083a8215e8d3a8165c2 P a033d9467d4d451586b0506fff8ebdf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a033d9467d4d451586b0506fff8ebdf9, State: Initialized, Role: FOLLOWER
I20260929 10:55:07.357057 19221 consensus_queue.cc:260] T 7056a1619d5a4083a8215e8d3a8165c2 P a033d9467d4d451586b0506fff8ebdf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } }
I20260929 10:55:07.357136 19221 ts_tablet_manager.cc:1428] T 7056a1619d5a4083a8215e8d3a8165c2 P a033d9467d4d451586b0506fff8ebdf9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:07.357604 19215 tablet_bootstrap.cc:492] T 7056a1619d5a4083a8215e8d3a8165c2 P 44a08b4b7d1c4435bee2560269289eb3: No bootstrap required, opened a new log
I20260929 10:55:07.357650 19215 ts_tablet_manager.cc:1397] T 7056a1619d5a4083a8215e8d3a8165c2 P 44a08b4b7d1c4435bee2560269289eb3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 10:55:07.357782 19215 raft_consensus.cc:348] T 7056a1619d5a4083a8215e8d3a8165c2 P 44a08b4b7d1c4435bee2560269289eb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } }
I20260929 10:55:07.357859 19215 raft_consensus.cc:374] T 7056a1619d5a4083a8215e8d3a8165c2 P 44a08b4b7d1c4435bee2560269289eb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:55:07.357879 19215 raft_consensus.cc:729] T 7056a1619d5a4083a8215e8d3a8165c2 P 44a08b4b7d1c4435bee2560269289eb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44a08b4b7d1c4435bee2560269289eb3, State: Initialized, Role: FOLLOWER
I20260929 10:55:07.357926 19215 consensus_queue.cc:260] T 7056a1619d5a4083a8215e8d3a8165c2 P 44a08b4b7d1c4435bee2560269289eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } }
I20260929 10:55:07.358008 19215 ts_tablet_manager.cc:1428] T 7056a1619d5a4083a8215e8d3a8165c2 P 44a08b4b7d1c4435bee2560269289eb3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:07.358256 19216 tablet_bootstrap.cc:492] T 7056a1619d5a4083a8215e8d3a8165c2 P 62f127242e214ef497866289454389c2: No bootstrap required, opened a new log
I20260929 10:55:07.358389 19216 ts_tablet_manager.cc:1397] T 7056a1619d5a4083a8215e8d3a8165c2 P 62f127242e214ef497866289454389c2: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260929 10:55:07.358563 19216 raft_consensus.cc:348] T 7056a1619d5a4083a8215e8d3a8165c2 P 62f127242e214ef497866289454389c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } }
I20260929 10:55:07.358655 19216 raft_consensus.cc:374] T 7056a1619d5a4083a8215e8d3a8165c2 P 62f127242e214ef497866289454389c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:55:07.358680 19216 raft_consensus.cc:729] T 7056a1619d5a4083a8215e8d3a8165c2 P 62f127242e214ef497866289454389c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62f127242e214ef497866289454389c2, State: Initialized, Role: FOLLOWER
I20260929 10:55:07.358734 19216 consensus_queue.cc:260] T 7056a1619d5a4083a8215e8d3a8165c2 P 62f127242e214ef497866289454389c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } }
I20260929 10:55:07.358819 19216 ts_tablet_manager.cc:1428] T 7056a1619d5a4083a8215e8d3a8165c2 P 62f127242e214ef497866289454389c2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:07.375495 19223 raft_consensus.cc:482] T 7056a1619d5a4083a8215e8d3a8165c2 P a033d9467d4d451586b0506fff8ebdf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:55:07.375583 19223 raft_consensus.cc:504] T 7056a1619d5a4083a8215e8d3a8165c2 P a033d9467d4d451586b0506fff8ebdf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } }
I20260929 10:55:07.375762 19223 leader_election.cc:288] T 7056a1619d5a4083a8215e8d3a8165c2 P a033d9467d4d451586b0506fff8ebdf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62f127242e214ef497866289454389c2 (127.10.238.193:34399), 44a08b4b7d1c4435bee2560269289eb3 (127.10.238.195:38935)
I20260929 10:55:07.375984 19140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7056a1619d5a4083a8215e8d3a8165c2" candidate_uuid: "a033d9467d4d451586b0506fff8ebdf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44a08b4b7d1c4435bee2560269289eb3" is_pre_election: true
I20260929 10:55:07.375983 18861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7056a1619d5a4083a8215e8d3a8165c2" candidate_uuid: "a033d9467d4d451586b0506fff8ebdf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62f127242e214ef497866289454389c2" is_pre_election: true
I20260929 10:55:07.376088 19140 raft_consensus.cc:2454] T 7056a1619d5a4083a8215e8d3a8165c2 P 44a08b4b7d1c4435bee2560269289eb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a033d9467d4d451586b0506fff8ebdf9 in term 0.
I20260929 10:55:07.376107 18861 raft_consensus.cc:2454] T 7056a1619d5a4083a8215e8d3a8165c2 P 62f127242e214ef497866289454389c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a033d9467d4d451586b0506fff8ebdf9 in term 0.
I20260929 10:55:07.376266 18934 leader_election.cc:302] T 7056a1619d5a4083a8215e8d3a8165c2 P a033d9467d4d451586b0506fff8ebdf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44a08b4b7d1c4435bee2560269289eb3, a033d9467d4d451586b0506fff8ebdf9; no voters:
I20260929 10:55:07.376354 19223 raft_consensus.cc:2781] T 7056a1619d5a4083a8215e8d3a8165c2 P a033d9467d4d451586b0506fff8ebdf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:55:07.376433 19223 raft_consensus.cc:482] T 7056a1619d5a4083a8215e8d3a8165c2 P a033d9467d4d451586b0506fff8ebdf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:55:07.376461 19223 raft_consensus.cc:3037] T 7056a1619d5a4083a8215e8d3a8165c2 P a033d9467d4d451586b0506fff8ebdf9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:55:07.377142 19223 raft_consensus.cc:504] T 7056a1619d5a4083a8215e8d3a8165c2 P a033d9467d4d451586b0506fff8ebdf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } }
I20260929 10:55:07.377305 19223 leader_election.cc:288] T 7056a1619d5a4083a8215e8d3a8165c2 P a033d9467d4d451586b0506fff8ebdf9 [CANDIDATE]: Term 1 election: Requested vote from peers 62f127242e214ef497866289454389c2 (127.10.238.193:34399), 44a08b4b7d1c4435bee2560269289eb3 (127.10.238.195:38935)
I20260929 10:55:07.377463 19140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7056a1619d5a4083a8215e8d3a8165c2" candidate_uuid: "a033d9467d4d451586b0506fff8ebdf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44a08b4b7d1c4435bee2560269289eb3"
I20260929 10:55:07.377458 18861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7056a1619d5a4083a8215e8d3a8165c2" candidate_uuid: "a033d9467d4d451586b0506fff8ebdf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62f127242e214ef497866289454389c2"
I20260929 10:55:07.377557 19140 raft_consensus.cc:3037] T 7056a1619d5a4083a8215e8d3a8165c2 P 44a08b4b7d1c4435bee2560269289eb3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:55:07.377560 18861 raft_consensus.cc:3037] T 7056a1619d5a4083a8215e8d3a8165c2 P 62f127242e214ef497866289454389c2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:55:07.378137 19140 raft_consensus.cc:2454] T 7056a1619d5a4083a8215e8d3a8165c2 P 44a08b4b7d1c4435bee2560269289eb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a033d9467d4d451586b0506fff8ebdf9 in term 1.
I20260929 10:55:07.378364 18934 leader_election.cc:302] T 7056a1619d5a4083a8215e8d3a8165c2 P a033d9467d4d451586b0506fff8ebdf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44a08b4b7d1c4435bee2560269289eb3, a033d9467d4d451586b0506fff8ebdf9; no voters:
I20260929 10:55:07.378461 19223 raft_consensus.cc:2781] T 7056a1619d5a4083a8215e8d3a8165c2 P a033d9467d4d451586b0506fff8ebdf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:55:07.378485 18861 raft_consensus.cc:2454] T 7056a1619d5a4083a8215e8d3a8165c2 P 62f127242e214ef497866289454389c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a033d9467d4d451586b0506fff8ebdf9 in term 1.
I20260929 10:55:07.378512 19223 raft_consensus.cc:686] T 7056a1619d5a4083a8215e8d3a8165c2 P a033d9467d4d451586b0506fff8ebdf9 [term 1 LEADER]: Becoming Leader. State: Replica: a033d9467d4d451586b0506fff8ebdf9, State: Running, Role: LEADER
I20260929 10:55:07.378567 19223 consensus_queue.cc:237] T 7056a1619d5a4083a8215e8d3a8165c2 P a033d9467d4d451586b0506fff8ebdf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } } peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } }
I20260929 10:55:07.379163 18703 catalog_manager.cc:5495] T 7056a1619d5a4083a8215e8d3a8165c2 P a033d9467d4d451586b0506fff8ebdf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a033d9467d4d451586b0506fff8ebdf9 (127.10.238.194). New cstate: current_term: 1 leader_uuid: "a033d9467d4d451586b0506fff8ebdf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a033d9467d4d451586b0506fff8ebdf9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 } health_report { overall_health: UNKNOWN } } }
I20260929 10:55:07.393630 19211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:07.390942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35326 (local address 127.10.238.194:35881)
0929 10:55:07.390980 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:07.390982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:07.390990 (+ 8us) server_negotiation.cc:407] Connection header received
0929 10:55:07.391068 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:07.391068 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:07.391077 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:07.391114 (+ 37us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:55:07.391300 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:07.391827 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:07.392916 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:07.393110 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:07.393170 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:07.393365 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:55:07.393398 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:55:07.393436 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:07.393534 (+ 98us) server_negotiation.cc:299] Negotiation successful
0929 10:55:07.393556 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 10:55:07.398031 19211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:07.395921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:59749 (local address 127.10.238.194:35881)
0929 10:55:07.395952 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:07.395954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:07.395962 (+ 8us) server_negotiation.cc:407] Connection header received
0929 10:55:07.395982 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:07.395982 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:07.395989 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:07.396048 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:55:07.396152 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:07.396602 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:07.397365 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:07.397547 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:07.397629 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:07.397776 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:55:07.397811 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:55:07.397870 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:07.397929 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 10:55:07.397948 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 10:55:07.398031 19269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:07.395758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.194:35881 (local address 127.10.238.194:59749)
0929 10:55:07.395823 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:07.395835 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:07.395881 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:07.396051 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:07.396052 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:07.396053 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 10:55:07.396093 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:07.396095 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:07.396683 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:07.396684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:07.397232 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:07.397233 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:07.397630 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:07.397630 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:07.397679 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:07.397686 (+ 7us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 10:55:07.397870 (+ 184us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 10:55:07.397872 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 10:55:07.397925 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 10:55:07.397966 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":2}
I20260929 10:55:07.400239 19140 raft_consensus.cc:1261] T 7056a1619d5a4083a8215e8d3a8165c2 P 44a08b4b7d1c4435bee2560269289eb3 [term 1 FOLLOWER]: Refusing update from remote peer a033d9467d4d451586b0506fff8ebdf9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:55:07.400259 18861 raft_consensus.cc:1261] T 7056a1619d5a4083a8215e8d3a8165c2 P 62f127242e214ef497866289454389c2 [term 1 FOLLOWER]: Refusing update from remote peer a033d9467d4d451586b0506fff8ebdf9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:55:07.400470 19223 consensus_queue.cc:1035] T 7056a1619d5a4083a8215e8d3a8165c2 P a033d9467d4d451586b0506fff8ebdf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44a08b4b7d1c4435bee2560269289eb3" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:55:07.400583 19225 consensus_queue.cc:1035] T 7056a1619d5a4083a8215e8d3a8165c2 P a033d9467d4d451586b0506fff8ebdf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62f127242e214ef497866289454389c2" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:55:07.413175 18704 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:55:07.413292 18704 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:55:07.413954 18704 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f5072e103f2c4e39a27e638aa276acb7: Sending DeleteTablet for 3 replicas of tablet 7056a1619d5a4083a8215e8d3a8165c2
I20260929 10:55:07.414166 18841 tablet_service.cc:1514] Processing DeleteTablet for tablet 7056a1619d5a4083a8215e8d3a8165c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:55:07 UTC) from {username='oryx', principal='oryx/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:59466
I20260929 10:55:07.414201 18981 tablet_service.cc:1514] Processing DeleteTablet for tablet 7056a1619d5a4083a8215e8d3a8165c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:55:07 UTC) from {username='oryx', principal='oryx/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:35296
I20260929 10:55:07.414280 19239 tablet_replica.cc:330] T 7056a1619d5a4083a8215e8d3a8165c2 P 62f127242e214ef497866289454389c2: stopping tablet replica
I20260929 10:55:07.414296 19121 tablet_service.cc:1514] Processing DeleteTablet for tablet 7056a1619d5a4083a8215e8d3a8165c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:55:07 UTC) from {username='oryx', principal='oryx/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:53794
I20260929 10:55:07.414314 19237 tablet_replica.cc:330] T 7056a1619d5a4083a8215e8d3a8165c2 P a033d9467d4d451586b0506fff8ebdf9: stopping tablet replica
I20260929 10:55:07.414335 19239 raft_consensus.cc:2229] T 7056a1619d5a4083a8215e8d3a8165c2 P 62f127242e214ef497866289454389c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:55:07.414371 19239 raft_consensus.cc:2258] T 7056a1619d5a4083a8215e8d3a8165c2 P 62f127242e214ef497866289454389c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:55:07.414363 19237 raft_consensus.cc:2229] T 7056a1619d5a4083a8215e8d3a8165c2 P a033d9467d4d451586b0506fff8ebdf9 [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:55:07.414427 19237 raft_consensus.cc:2258] T 7056a1619d5a4083a8215e8d3a8165c2 P a033d9467d4d451586b0506fff8ebdf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:55:07.414505 19238 tablet_replica.cc:330] T 7056a1619d5a4083a8215e8d3a8165c2 P 44a08b4b7d1c4435bee2560269289eb3: stopping tablet replica
I20260929 10:55:07.414559 19238 raft_consensus.cc:2229] T 7056a1619d5a4083a8215e8d3a8165c2 P 44a08b4b7d1c4435bee2560269289eb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:55:07.414593 19238 raft_consensus.cc:2258] T 7056a1619d5a4083a8215e8d3a8165c2 P 44a08b4b7d1c4435bee2560269289eb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:55:07.414891 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 18776
I20260929 10:55:07.416718 19237 ts_tablet_manager.cc:1905] T 7056a1619d5a4083a8215e8d3a8165c2 P a033d9467d4d451586b0506fff8ebdf9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:55:07.418447 19237 ts_tablet_manager.cc:1918] T 7056a1619d5a4083a8215e8d3a8165c2 P a033d9467d4d451586b0506fff8ebdf9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 10:55:07.418529 19237 log.cc:1196] T 7056a1619d5a4083a8215e8d3a8165c2 P a033d9467d4d451586b0506fff8ebdf9: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-1/wal/wals/7056a1619d5a4083a8215e8d3a8165c2
I20260929 10:55:07.418797 19237 ts_tablet_manager.cc:1939] T 7056a1619d5a4083a8215e8d3a8165c2 P a033d9467d4d451586b0506fff8ebdf9: Deleting consensus metadata
I20260929 10:55:07.419186 18690 catalog_manager.cc:4841] TS a033d9467d4d451586b0506fff8ebdf9 (127.10.238.194:35881): tablet 7056a1619d5a4083a8215e8d3a8165c2 (table test-table [id=9aa903b2d47f4d33bb0aae38dadbfde1]) successfully deleted
I20260929 10:55:07.421547 19238 ts_tablet_manager.cc:1905] T 7056a1619d5a4083a8215e8d3a8165c2 P 44a08b4b7d1c4435bee2560269289eb3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 10:55:07.421566 18689 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.238.193:34399 (error 108)
I20260929 10:55:07.421813 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 18916
I20260929 10:55:07.423323 19238 ts_tablet_manager.cc:1918] T 7056a1619d5a4083a8215e8d3a8165c2 P 44a08b4b7d1c4435bee2560269289eb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 10:55:07.423381 19238 log.cc:1196] T 7056a1619d5a4083a8215e8d3a8165c2 P 44a08b4b7d1c4435bee2560269289eb3: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790679274510869-11195-0/minicluster-data/ts-2/wal/wals/7056a1619d5a4083a8215e8d3a8165c2
I20260929 10:55:07.423628 19238 ts_tablet_manager.cc:1939] T 7056a1619d5a4083a8215e8d3a8165c2 P 44a08b4b7d1c4435bee2560269289eb3: Deleting consensus metadata
I20260929 10:55:07.423996 18690 catalog_manager.cc:4841] TS 44a08b4b7d1c4435bee2560269289eb3 (127.10.238.195:38935): tablet 7056a1619d5a4083a8215e8d3a8165c2 (table test-table [id=9aa903b2d47f4d33bb0aae38dadbfde1]) successfully deleted
I20260929 10:55:07.424292 19304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:07.424123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:34399 (local address 127.0.0.1:59480)
0929 10:55:07.424214 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:07.424258 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.193:34399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73,"thread_start_us":52,"threads_started":1}
W20260929 10:55:07.424437 18689 catalog_manager.cc:4584] TS 62f127242e214ef497866289454389c2 (127.10.238.193:34399): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7056a1619d5a4083a8215e8d3a8165c2: Network error: Client connection negotiation failed: client connection to 127.10.238.193:34399: connect: Connection refused (error 111)
I20260929 10:55:07.429389 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 19056
I20260929 10:55:07.435478 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 18672
2026-09-29T10:55:07Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2630 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790679274510869-11195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:55:07 dist-test-slave-8nrt krb5kdc[19307](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:55:07 dist-test-slave-8nrt krb5kdc[19307](info): set up 2 sockets
Sep 29 10:55:07 dist-test-slave-8nrt krb5kdc[19307](info): commencing operation
krb5kdc: starting...
W20260929 10:55:08.477970 11195 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Sep 29 10:55:08 dist-test-slave-8nrt krb5kdc[19307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:55:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:55:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
I20260929 10:55:08.596974 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:46167
--webserver_interface=127.10.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:43861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:46167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:55:08.681346 19320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:55:08.681603 19320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:55:08.681644 19320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:55:08.682811 19320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:55:08.682884 19320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:55:08.682912 19320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:55:08.682945 19320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:55:08.682971 19320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:55:08.684448 19320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:43861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790679274510869-11195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:46167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:46167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:55:08.684743 19320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:55:08.684975 19320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:55:08.690038 19325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:08.690033 19328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:08.690035 19326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:55:08.690326 19320 server_base.cc:1034] running on GCE node
I20260929 10:55:08.690551 19320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:55:08.690800 19320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:55:08.691944 19320 hybrid_clock.cc:648] HybridClock initialized: now 1790679308691925 us; error 41 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T10:55:08Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1258 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:55:08 dist-test-slave-8nrt krb5kdc[19333](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:55:08 dist-test-slave-8nrt krb5kdc[19333](info): set up 2 sockets
Sep 29 10:55:08 dist-test-slave-8nrt krb5kdc[19333](info): commencing operation
krb5kdc: starting...
W20260929 10:55:09.734031 11195 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Sep 29 10:55:09 dist-test-slave-8nrt krb5kdc[19333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:55:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:55:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
I20260929 10:55:09.849490 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:43847
--webserver_interface=127.10.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:45655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:43847
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:55:09.929302 19346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:55:09.929512 19346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:55:09.929534 19346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:55:09.930960 19346 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:55:09.931020 19346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:55:09.931057 19346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:55:09.931080 19346 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:55:09.931151 19346 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:55:09.932812 19346 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:45655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:43847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:43847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:55:09.933022 19346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:55:09.933182 19346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:55:09.937992 19351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:09.938054 19352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:09.938086 19354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:55:09.938470 19346 server_base.cc:1034] running on GCE node
I20260929 10:55:09.938716 19346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:55:09.939051 19346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:55:09.940217 19346 hybrid_clock.cc:648] HybridClock initialized: now 1790679309940200 us; error 44 us; skew 500 ppm
Sep 29 10:55:09 dist-test-slave-8nrt krb5kdc[19333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679309, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:55:09.947005 19346 init.cc:377] Logged in from keytab as kudu/127.10.238.254@KRBTEST.COM (short username kudu)
I20260929 10:55:09.947510 19346 webserver.cc:460] Webserver started at http://127.10.238.254:43665/ using document root <none> and password file <none>
I20260929 10:55:09.947762 19346 fs_manager.cc:359] Metadata directory not provided
I20260929 10:55:09.947834 19346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:55:09.947989 19346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:55:09.949083 19346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/master-0/data/instance:
uuid: "8d95325ad5934ff3b656fdd49694d615"
format_stamp: "Formatted at 2026-09-29 10:55:09 on dist-test-slave-8nrt"
server_key: "8be3062b04e143655562ef938f38f630"
server_key_iv: "7892c513f964182c0f895c7770d1cc7c"
server_key_version: "encryptionkey@0"
I20260929 10:55:09.949491 19346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/master-0/wal/instance:
uuid: "8d95325ad5934ff3b656fdd49694d615"
format_stamp: "Formatted at 2026-09-29 10:55:09 on dist-test-slave-8nrt"
server_key: "8be3062b04e143655562ef938f38f630"
server_key_iv: "7892c513f964182c0f895c7770d1cc7c"
server_key_version: "encryptionkey@0"
I20260929 10:55:09.950862 19346 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:55:09.951597 19361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:09.951787 19346 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:55:09.951881 19346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "8d95325ad5934ff3b656fdd49694d615"
format_stamp: "Formatted at 2026-09-29 10:55:09 on dist-test-slave-8nrt"
server_key: "8be3062b04e143655562ef938f38f630"
server_key_iv: "7892c513f964182c0f895c7770d1cc7c"
server_key_version: "encryptionkey@0"
I20260929 10:55:09.951938 19346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:55:09.986186 19346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:55:09.988581 19346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:55:09.988700 19346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:55:09.992760 19346 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.254:43847
I20260929 10:55:09.992800 19413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.254:43847 every 8 connection(s)
I20260929 10:55:09.993187 19346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:55:09.994251 19414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:55:09.994264 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 19346
I20260929 10:55:09.994380 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/master-0/wal/instance
I20260929 10:55:09.994603 11195 external_mini_cluster.cc:1428] Setting key a1c92c012ecb694f7f48c5b9a512dc1a
I20260929 10:55:09.997727 19414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d95325ad5934ff3b656fdd49694d615: Bootstrap starting.
I20260929 10:55:09.998413 19414 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d95325ad5934ff3b656fdd49694d615: Neither blocks nor log segments found. Creating new log.
I20260929 10:55:09.998713 19414 log.cc:824] T 00000000000000000000000000000000 P 8d95325ad5934ff3b656fdd49694d615: Log is configured to *not* fsync() on all Append() calls
I20260929 10:55:09.999399 19414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d95325ad5934ff3b656fdd49694d615: No bootstrap required, opened a new log
I20260929 10:55:10.001168 19414 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d95325ad5934ff3b656fdd49694d615 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d95325ad5934ff3b656fdd49694d615" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 43847 } }
Sep 29 10:55:10 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:55:10.001395 19414 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d95325ad5934ff3b656fdd49694d615 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:55:10.001449 19414 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d95325ad5934ff3b656fdd49694d615 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d95325ad5934ff3b656fdd49694d615, State: Initialized, Role: FOLLOWER
I20260929 10:55:10.001590 19414 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d95325ad5934ff3b656fdd49694d615 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d95325ad5934ff3b656fdd49694d615" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 43847 } }
I20260929 10:55:10.001688 19414 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d95325ad5934ff3b656fdd49694d615 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:55:10.001725 19414 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d95325ad5934ff3b656fdd49694d615 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:55:10.001775 19414 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d95325ad5934ff3b656fdd49694d615 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:55:10.002337 19414 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d95325ad5934ff3b656fdd49694d615 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d95325ad5934ff3b656fdd49694d615" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 43847 } }
I20260929 10:55:10.002426 19414 leader_election.cc:302] T 00000000000000000000000000000000 P 8d95325ad5934ff3b656fdd49694d615 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d95325ad5934ff3b656fdd49694d615; no voters:
I20260929 10:55:10.002633 19414 leader_election.cc:288] T 00000000000000000000000000000000 P 8d95325ad5934ff3b656fdd49694d615 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:55:10.002661 19419 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d95325ad5934ff3b656fdd49694d615 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:55:10.002779 19419 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d95325ad5934ff3b656fdd49694d615 [term 1 LEADER]: Becoming Leader. State: Replica: 8d95325ad5934ff3b656fdd49694d615, State: Running, Role: LEADER
I20260929 10:55:10.002862 19419 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d95325ad5934ff3b656fdd49694d615 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d95325ad5934ff3b656fdd49694d615" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 43847 } }
I20260929 10:55:10.002905 19414 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d95325ad5934ff3b656fdd49694d615 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:55:10.003188 19419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d95325ad5934ff3b656fdd49694d615 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d95325ad5934ff3b656fdd49694d615" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d95325ad5934ff3b656fdd49694d615" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 43847 } } }
I20260929 10:55:10.003259 19419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d95325ad5934ff3b656fdd49694d615 [sys.catalog]: This master's current role is: LEADER
I20260929 10:55:10.003384 19419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d95325ad5934ff3b656fdd49694d615 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d95325ad5934ff3b656fdd49694d615. Latest consensus state: current_term: 1 leader_uuid: "8d95325ad5934ff3b656fdd49694d615" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d95325ad5934ff3b656fdd49694d615" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 43847 } } }
I20260929 10:55:10.003465 19419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d95325ad5934ff3b656fdd49694d615 [sys.catalog]: This master's current role is: LEADER
I20260929 10:55:10.003598 19424 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:55:10.004179 19424 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:55:10.006093 19417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:09.995159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50862 (local address 127.10.238.254:43847)
0929 10:55:09.995345 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:09.995355 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:09.995389 (+ 34us) server_negotiation.cc:407] Connection header received
0929 10:55:09.995910 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:09.995920 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:09.995986 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:09.996143 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:09.996755 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:09.998168 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:09.998986 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:09.999401 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:09.999519 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:10.001985 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:10.001988 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:10.001989 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:10.001999 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:10.004432 (+ 2433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:10.004856 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:10.004859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:10.004859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:10.004960 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:10.005161 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:10.005162 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:10.005162 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:10.005483 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:10.005629 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:10.005749 (+ 120us) server_negotiation.cc:299] Negotiation successful
0929 10:55:10.005897 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":76,"threads_started":1}
I20260929 10:55:10.006261 19424 catalog_manager.cc:1347] Generated new cluster ID: 721d27f514884c39b199a7352fa6ca90
I20260929 10:55:10.006444 19424 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:55:10.019891 19424 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:55:10.020437 19424 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:55:10.024297 19424 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d95325ad5934ff3b656fdd49694d615: Generated new TSK 0
I20260929 10:55:10.024477 19424 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.238.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab.
Entry for principal kudu/127.10.238.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab.
I20260929 10:55:10.060120 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.193:0
--local_ip_for_outbound_sockets=127.10.238.193
--webserver_interface=127.10.238.193
--webserver_port=0
--tserver_master_addrs=127.10.238.254:43847
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab
--principal=kudu/127.10.238.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:45655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:55:10.141702 19440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:55:10.141919 19440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:55:10.141944 19440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:55:10.143146 19440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:55:10.143189 19440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:55:10.143216 19440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.193
I20260929 10:55:10.144745 19440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:45655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab
--principal=kudu/127.10.238.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.238.193
--webserver_port=0
--tserver_master_addrs=127.10.238.254:43847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.193
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:55:10.145002 19440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:55:10.145242 19440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:55:10.150408 19448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:55:10.150578 19440 server_base.cc:1034] running on GCE node
W20260929 10:55:10.150389 19445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:10.150394 19446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:55:10.150947 19440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:55:10.151230 19440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:55:10.152395 19440 hybrid_clock.cc:648] HybridClock initialized: now 1790679310152379 us; error 41 us; skew 500 ppm
Sep 29 10:55:10 dist-test-slave-8nrt krb5kdc[19333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679310, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:55:10.155287 19440 init.cc:377] Logged in from keytab as kudu/127.10.238.193@KRBTEST.COM (short username kudu)
I20260929 10:55:10.155752 19440 webserver.cc:460] Webserver started at http://127.10.238.193:45657/ using document root <none> and password file <none>
I20260929 10:55:10.155944 19440 fs_manager.cc:359] Metadata directory not provided
I20260929 10:55:10.155985 19440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:55:10.156075 19440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:55:10.157095 19440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-0/data/instance:
uuid: "081708d53308419db3732c5e5d5cefc9"
format_stamp: "Formatted at 2026-09-29 10:55:10 on dist-test-slave-8nrt"
server_key: "73d8f28ec27a2411a12ac6e68ae924dc"
server_key_iv: "f6b3a8d0f8ae4f4bdac0142a639dd582"
server_key_version: "encryptionkey@0"
I20260929 10:55:10.157476 19440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-0/wal/instance:
uuid: "081708d53308419db3732c5e5d5cefc9"
format_stamp: "Formatted at 2026-09-29 10:55:10 on dist-test-slave-8nrt"
server_key: "73d8f28ec27a2411a12ac6e68ae924dc"
server_key_iv: "f6b3a8d0f8ae4f4bdac0142a639dd582"
server_key_version: "encryptionkey@0"
I20260929 10:55:10.158782 19440 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 10:55:10.159508 19455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:10.159710 19440 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 10:55:10.159785 19440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-0/wal
uuid: "081708d53308419db3732c5e5d5cefc9"
format_stamp: "Formatted at 2026-09-29 10:55:10 on dist-test-slave-8nrt"
server_key: "73d8f28ec27a2411a12ac6e68ae924dc"
server_key_iv: "f6b3a8d0f8ae4f4bdac0142a639dd582"
server_key_version: "encryptionkey@0"
I20260929 10:55:10.159886 19440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:55:10.168910 19440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:55:10.171363 19440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:55:10.171516 19440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:55:10.171797 19440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:55:10.172125 19440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:55:10.172159 19440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:10.172181 19440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:55:10.172216 19440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:10.178527 19440 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.193:40321
I20260929 10:55:10.178570 19568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.193:40321 every 8 connection(s)
I20260929 10:55:10.178901 19440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
I20260929 10:55:10.184958 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 19440
Sep 29 10:55:10 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679310, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.193@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:55:10.185045 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-0/wal/instance
I20260929 10:55:10.185317 11195 external_mini_cluster.cc:1428] Setting key 59f2d8a4e8500e3b8b00eccca0c30ef6
I20260929 10:55:10.188557 19417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:10.179635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:44947 (local address 127.10.238.254:43847)
0929 10:55:10.179707 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:10.179709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:10.179941 (+ 232us) server_negotiation.cc:407] Connection header received
0929 10:55:10.180427 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:10.180428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:10.180439 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:10.180497 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:10.181536 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:10.182052 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:10.183027 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:10.183225 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:10.183281 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:10.185654 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:10.185655 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:10.185655 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:10.185663 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:10.187119 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:10.187559 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:10.187561 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:10.187561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:10.187622 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:10.187892 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:10.187893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:10.187893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:10.188011 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:10.188083 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:10.188378 (+ 295us) server_negotiation.cc:299] Negotiation successful
0929 10:55:10.188466 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 10:55:10.188759 19571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:10.179629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.193:44947)
0929 10:55:10.179806 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:10.179844 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:10.180344 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:10.180573 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:10.180576 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:10.181152 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:10.181450 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:10.181456 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:10.182137 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:10.182137 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:10.182948 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:10.182951 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:10.183310 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:10.183311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:10.183410 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:10.183967 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:10.183975 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:10.185553 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:55:10.187226 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:10.187234 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:10.187236 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:10.187467 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:10.187692 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:10.187693 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:10.187694 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:10.187816 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:10.188086 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:55:10.188094 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:55:10.188238 (+ 144us) client_negotiation.cc:769] Sending connection context
0929 10:55:10.188355 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 10:55:10.188516 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":111,"threads_started":1}
I20260929 10:55:10.189494 19569 heartbeater.cc:347] Connected to a master server at 127.10.238.254:43847
I20260929 10:55:10.189594 19569 heartbeater.cc:464] Registering TS with master...
I20260929 10:55:10.189811 19569 heartbeater.cc:511] Master 127.10.238.254:43847 requested a full tablet report, sending...
I20260929 10:55:10.190385 19378 ts_manager.cc:194] Registered new tserver with Master: 081708d53308419db3732c5e5d5cefc9 (127.10.238.193:40321)
I20260929 10:55:10.191403 19378 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.193@KRBTEST.COM'} at 127.10.238.193:44947
WARNING: no policy specified for kudu/127.10.238.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab.
Entry for principal kudu/127.10.238.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab.
I20260929 10:55:10.212361 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.194:0
--local_ip_for_outbound_sockets=127.10.238.194
--webserver_interface=127.10.238.194
--webserver_port=0
--tserver_master_addrs=127.10.238.254:43847
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab
--principal=kudu/127.10.238.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:45655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:55:10.294003 19574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:55:10.294203 19574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:55:10.294229 19574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:55:10.295336 19574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:55:10.295369 19574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:55:10.295395 19574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.194
I20260929 10:55:10.296761 19574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:45655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab
--principal=kudu/127.10.238.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.238.194
--webserver_port=0
--tserver_master_addrs=127.10.238.254:43847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.194
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:55:10.296934 19574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:55:10.297120 19574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:55:10.302078 19580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:10.302145 19579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:10.302078 19582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:55:10.302389 19574 server_base.cc:1034] running on GCE node
I20260929 10:55:10.302659 19574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:55:10.302863 19574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:55:10.303998 19574 hybrid_clock.cc:648] HybridClock initialized: now 1790679310303985 us; error 27 us; skew 500 ppm
Sep 29 10:55:10 dist-test-slave-8nrt krb5kdc[19333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679310, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:55:10.306561 19574 init.cc:377] Logged in from keytab as kudu/127.10.238.194@KRBTEST.COM (short username kudu)
I20260929 10:55:10.307071 19574 webserver.cc:460] Webserver started at http://127.10.238.194:37707/ using document root <none> and password file <none>
I20260929 10:55:10.307258 19574 fs_manager.cc:359] Metadata directory not provided
I20260929 10:55:10.307299 19574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:55:10.307389 19574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:55:10.308477 19574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-1/data/instance:
uuid: "a3ed66e3e60b4a92b331ce5a24645e07"
format_stamp: "Formatted at 2026-09-29 10:55:10 on dist-test-slave-8nrt"
server_key: "5da713172568d08abe42d0642be5a550"
server_key_iv: "ebd358e3a50bd57f081d7f87a14618f9"
server_key_version: "encryptionkey@0"
I20260929 10:55:10.308792 19574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-1/wal/instance:
uuid: "a3ed66e3e60b4a92b331ce5a24645e07"
format_stamp: "Formatted at 2026-09-29 10:55:10 on dist-test-slave-8nrt"
server_key: "5da713172568d08abe42d0642be5a550"
server_key_iv: "ebd358e3a50bd57f081d7f87a14618f9"
server_key_version: "encryptionkey@0"
I20260929 10:55:10.310154 19574 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 10:55:10.310884 19589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:10.311084 19574 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 10:55:10.311172 19574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-1/wal
uuid: "a3ed66e3e60b4a92b331ce5a24645e07"
format_stamp: "Formatted at 2026-09-29 10:55:10 on dist-test-slave-8nrt"
server_key: "5da713172568d08abe42d0642be5a550"
server_key_iv: "ebd358e3a50bd57f081d7f87a14618f9"
server_key_version: "encryptionkey@0"
I20260929 10:55:10.311226 19574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:55:10.336005 19574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:55:10.338495 19574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:55:10.338616 19574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:55:10.338807 19574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:55:10.339133 19574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:55:10.339177 19574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:10.339200 19574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:55:10.339249 19574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:10.345047 19574 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.194:43849
I20260929 10:55:10.345130 19702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.194:43849 every 8 connection(s)
I20260929 10:55:10.345484 19574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
I20260929 10:55:10.347451 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 19574
I20260929 10:55:10.347602 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-1/wal/instance
I20260929 10:55:10.347848 11195 external_mini_cluster.cc:1428] Setting key 778d393d0f42faa09468fa4e01cf8f7a
Sep 29 10:55:10 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679310, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.194@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.238.195@KRBTEST.COM; defaulting to no policy
I20260929 10:55:10.356117 19417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:10.346247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:38529 (local address 127.10.238.254:43847)
0929 10:55:10.346316 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:10.346319 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:10.346561 (+ 242us) server_negotiation.cc:407] Connection header received
0929 10:55:10.347089 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:10.347090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:10.347101 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:10.347157 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:10.348533 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:10.349069 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:10.350209 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:10.350424 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:10.350510 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:10.353029 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:10.353031 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:10.353032 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:10.353040 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:10.354604 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:10.355049 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:10.355050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:10.355051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:10.355114 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:10.355390 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:10.355391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:10.355391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:10.355535 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:10.355613 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:10.355942 (+ 329us) server_negotiation.cc:299] Negotiation successful
0929 10:55:10.356023 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 10:55:10.356338 19705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:10.346241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.194:38529)
0929 10:55:10.346424 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:10.346461 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:10.346995 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:10.347258 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:10.347263 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:10.348137 (+ 874us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:10.348423 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:10.348430 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:10.349189 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:10.349190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:10.350086 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:10.350089 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:10.350524 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:10.350525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:10.350668 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:10.351233 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:10.351242 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:10.352903 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:55:10.354711 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:10.354718 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:10.354721 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:10.354954 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:10.355185 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:10.355186 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:10.355187 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:10.355307 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:10.355605 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:55:10.355612 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:55:10.355776 (+ 164us) client_negotiation.cc:769] Sending connection context
0929 10:55:10.355912 (+ 136us) client_negotiation.cc:240] Negotiation successful
0929 10:55:10.356080 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":104,"threads_started":1}
I20260929 10:55:10.356886 19703 heartbeater.cc:347] Connected to a master server at 127.10.238.254:43847
I20260929 10:55:10.357039 19703 heartbeater.cc:464] Registering TS with master...
I20260929 10:55:10.357295 19703 heartbeater.cc:511] Master 127.10.238.254:43847 requested a full tablet report, sending...
I20260929 10:55:10.357689 19378 ts_manager.cc:194] Registered new tserver with Master: a3ed66e3e60b4a92b331ce5a24645e07 (127.10.238.194:43849)
I20260929 10:55:10.358052 19378 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.194@KRBTEST.COM'} at 127.10.238.194:38529
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab.
Entry for principal kudu/127.10.238.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab.
I20260929 10:55:10.374591 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.195:0
--local_ip_for_outbound_sockets=127.10.238.195
--webserver_interface=127.10.238.195
--webserver_port=0
--tserver_master_addrs=127.10.238.254:43847
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab
--principal=kudu/127.10.238.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:45655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:55:10.456399 19708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:55:10.456610 19708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:55:10.456634 19708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:55:10.457820 19708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:55:10.457858 19708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:55:10.457885 19708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.195
I20260929 10:55:10.459448 19708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:45655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab
--principal=kudu/127.10.238.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.238.195
--webserver_port=0
--tserver_master_addrs=127.10.238.254:43847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.195
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:55:10.459647 19708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:55:10.459846 19708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:55:10.465004 19716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:10.465004 19713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:10.465039 19714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:55:10.465082 19708 server_base.cc:1034] running on GCE node
I20260929 10:55:10.465441 19708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:55:10.465652 19708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:55:10.466800 19708 hybrid_clock.cc:648] HybridClock initialized: now 1790679310466783 us; error 40 us; skew 500 ppm
Sep 29 10:55:10 dist-test-slave-8nrt krb5kdc[19333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679310, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:55:10.469378 19708 init.cc:377] Logged in from keytab as kudu/127.10.238.195@KRBTEST.COM (short username kudu)
I20260929 10:55:10.469866 19708 webserver.cc:460] Webserver started at http://127.10.238.195:40837/ using document root <none> and password file <none>
I20260929 10:55:10.470117 19708 fs_manager.cc:359] Metadata directory not provided
I20260929 10:55:10.470187 19708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:55:10.470345 19708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:55:10.471408 19708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-2/data/instance:
uuid: "d7f8ed4b94024663bdf2477437955545"
format_stamp: "Formatted at 2026-09-29 10:55:10 on dist-test-slave-8nrt"
server_key: "17df44e19bf3762b475a4623fef5c2e3"
server_key_iv: "ab27c11a0fc2d4c50d935654168b1751"
server_key_version: "encryptionkey@0"
I20260929 10:55:10.471782 19708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-2/wal/instance:
uuid: "d7f8ed4b94024663bdf2477437955545"
format_stamp: "Formatted at 2026-09-29 10:55:10 on dist-test-slave-8nrt"
server_key: "17df44e19bf3762b475a4623fef5c2e3"
server_key_iv: "ab27c11a0fc2d4c50d935654168b1751"
server_key_version: "encryptionkey@0"
I20260929 10:55:10.473155 19708 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 10:55:10.473964 19723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:10.474179 19708 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 10:55:10.474308 19708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-2/wal
uuid: "d7f8ed4b94024663bdf2477437955545"
format_stamp: "Formatted at 2026-09-29 10:55:10 on dist-test-slave-8nrt"
server_key: "17df44e19bf3762b475a4623fef5c2e3"
server_key_iv: "ab27c11a0fc2d4c50d935654168b1751"
server_key_version: "encryptionkey@0"
I20260929 10:55:10.474386 19708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:55:10.498199 19708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:55:10.500264 19708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:55:10.500427 19708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:55:10.500663 19708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:55:10.500985 19708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:55:10.501040 19708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:10.501082 19708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:55:10.501111 19708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:10.507858 19708 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.195:44805
I20260929 10:55:10.507899 19836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.195:44805 every 8 connection(s)
I20260929 10:55:10.508247 19708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
I20260929 10:55:10.509757 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 19708
I20260929 10:55:10.509857 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-2/wal/instance
I20260929 10:55:10.510041 11195 external_mini_cluster.cc:1428] Setting key 3df56ecbb1d95c016d706c09d4dfe8c9
Sep 29 10:55:10 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679310, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.195@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:55:10.517598 19417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:10.509086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:49367 (local address 127.10.238.254:43847)
0929 10:55:10.509129 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:10.509131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:10.509345 (+ 214us) server_negotiation.cc:407] Connection header received
0929 10:55:10.510126 (+ 781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:10.510128 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:10.510141 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:10.510190 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:10.511385 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:10.511878 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:10.512823 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:10.513001 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:10.513050 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:10.515328 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:10.515329 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:10.515329 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:10.515336 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:10.516458 (+ 1122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:10.516811 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:10.516812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:10.516813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:10.516861 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:10.517031 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:10.517032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:10.517032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:10.517128 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:10.517177 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:10.517474 (+ 297us) server_negotiation.cc:299] Negotiation successful
0929 10:55:10.517536 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 10:55:10.517759 19839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:10.509051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.195:49367)
0929 10:55:10.509190 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:10.509235 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:10.509979 (+ 744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:10.510284 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:10.510288 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:10.511107 (+ 819us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:10.511308 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:10.511312 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:10.511956 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:10.511957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:10.512755 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:10.512756 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:10.513070 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:10.513070 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:10.513169 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:10.513791 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:10.513799 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:10.515251 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:55:10.516525 (+ 1274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:10.516531 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:10.516532 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:10.516748 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:10.516900 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:10.516901 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:10.516901 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:10.516983 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:10.517189 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:55:10.517194 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:55:10.517363 (+ 169us) client_negotiation.cc:769] Sending connection context
0929 10:55:10.517447 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 10:55:10.517562 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":65,"threads_started":1}
I20260929 10:55:10.518251 19837 heartbeater.cc:347] Connected to a master server at 127.10.238.254:43847
I20260929 10:55:10.518362 19837 heartbeater.cc:464] Registering TS with master...
I20260929 10:55:10.518532 19837 heartbeater.cc:511] Master 127.10.238.254:43847 requested a full tablet report, sending...
I20260929 10:55:10.518923 19378 ts_manager.cc:194] Registered new tserver with Master: d7f8ed4b94024663bdf2477437955545 (127.10.238.195:44805)
I20260929 10:55:10.519296 19378 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.195@KRBTEST.COM'} at 127.10.238.195:49367
I20260929 10:55:10.521791 11195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.10.238.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.238.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.238.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.193.keytab.
Entry for principal oryx/127.10.238.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.193.keytab.
WARNING: no policy specified for oryx/127.10.238.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.238.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.238.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.194.keytab.
Entry for principal oryx/127.10.238.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.194.keytab.
WARNING: no policy specified for oryx/127.10.238.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.238.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.238.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.195.keytab.
Entry for principal oryx/127.10.238.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.195.keytab.
I20260929 10:55:10.592805 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 19440
I20260929 10:55:10.597993 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 19574
I20260929 10:55:10.602577 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 19708
I20260929 10:55:10.606839 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 19346
I20260929 10:55:10.612166 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:43847
--webserver_interface=127.10.238.254
--webserver_port=43665
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:45655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:43847
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:55:10.693169 19846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:55:10.693460 19846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:55:10.693502 19846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:55:10.694679 19846 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:55:10.694752 19846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:55:10.694779 19846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:55:10.694811 19846 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:55:10.694837 19846 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:55:10.696350 19846 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:45655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:43847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:43847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.238.254
--webserver_port=43665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:55:10.696638 19846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:55:10.696877 19846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:55:10.701930 19851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:10.701900 19854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:55:10.701932 19846 server_base.cc:1034] running on GCE node
W20260929 10:55:10.701910 19852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:55:10.702396 19846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:55:10.702765 19846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:55:10.703922 19846 hybrid_clock.cc:648] HybridClock initialized: now 1790679310703899 us; error 39 us; skew 500 ppm
Sep 29 10:55:10 dist-test-slave-8nrt krb5kdc[19333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679310, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:55:10.706580 19846 init.cc:377] Logged in from keytab as kudu/127.10.238.254@KRBTEST.COM (short username kudu)
I20260929 10:55:10.707049 19846 webserver.cc:460] Webserver started at http://127.10.238.254:43665/ using document root <none> and password file <none>
I20260929 10:55:10.707315 19846 fs_manager.cc:359] Metadata directory not provided
I20260929 10:55:10.707408 19846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:55:10.708794 19846 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:55:10.709681 19861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:10.709867 19846 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 10:55:10.709980 19846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "8d95325ad5934ff3b656fdd49694d615"
format_stamp: "Formatted at 2026-09-29 10:55:09 on dist-test-slave-8nrt"
server_key: "8be3062b04e143655562ef938f38f630"
server_key_iv: "7892c513f964182c0f895c7770d1cc7c"
server_key_version: "encryptionkey@0"
I20260929 10:55:10.710367 19846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:55:10.754760 19846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:55:10.757727 19846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:55:10.757937 19846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:55:10.762521 19846 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.254:43847
I20260929 10:55:10.762558 19913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.254:43847 every 8 connection(s)
I20260929 10:55:10.763029 19846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:55:10.764544 19914 sys_catalog.cc:263] Verifying existing consensus state
I20260929 10:55:10.765337 19914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d95325ad5934ff3b656fdd49694d615: Bootstrap starting.
I20260929 10:55:10.767509 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 19846
I20260929 10:55:10.767851 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.193:40321
--local_ip_for_outbound_sockets=127.10.238.193
--tserver_master_addrs=127.10.238.254:43847
--webserver_port=45657
--webserver_interface=127.10.238.193
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab
--principal=kudu/127.10.238.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:45655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.10.238.193
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:55:10.774686 19914 log.cc:824] T 00000000000000000000000000000000 P 8d95325ad5934ff3b656fdd49694d615: Log is configured to *not* fsync() on all Append() calls
I20260929 10:55:10.776409 19914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d95325ad5934ff3b656fdd49694d615: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 10:55:10.776721 19914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d95325ad5934ff3b656fdd49694d615: Bootstrap complete.
I20260929 10:55:10.779136 19914 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d95325ad5934ff3b656fdd49694d615 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d95325ad5934ff3b656fdd49694d615" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 43847 } }
I20260929 10:55:10.779373 19914 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d95325ad5934ff3b656fdd49694d615 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d95325ad5934ff3b656fdd49694d615, State: Initialized, Role: FOLLOWER
I20260929 10:55:10.779546 19914 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d95325ad5934ff3b656fdd49694d615 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d95325ad5934ff3b656fdd49694d615" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 43847 } }
I20260929 10:55:10.779618 19914 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d95325ad5934ff3b656fdd49694d615 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:55:10.779668 19914 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d95325ad5934ff3b656fdd49694d615 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:55:10.779717 19914 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d95325ad5934ff3b656fdd49694d615 [term 1 FOLLOWER]: Advancing to term 2
I20260929 10:55:10.780710 19914 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d95325ad5934ff3b656fdd49694d615 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d95325ad5934ff3b656fdd49694d615" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 43847 } }
I20260929 10:55:10.780798 19914 leader_election.cc:302] T 00000000000000000000000000000000 P 8d95325ad5934ff3b656fdd49694d615 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d95325ad5934ff3b656fdd49694d615; no voters:
I20260929 10:55:10.781032 19914 leader_election.cc:288] T 00000000000000000000000000000000 P 8d95325ad5934ff3b656fdd49694d615 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 10:55:10.781075 19918 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d95325ad5934ff3b656fdd49694d615 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 10:55:10.781224 19918 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d95325ad5934ff3b656fdd49694d615 [term 2 LEADER]: Becoming Leader. State: Replica: 8d95325ad5934ff3b656fdd49694d615, State: Running, Role: LEADER
I20260929 10:55:10.781347 19914 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d95325ad5934ff3b656fdd49694d615 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:55:10.781391 19918 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d95325ad5934ff3b656fdd49694d615 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d95325ad5934ff3b656fdd49694d615" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 43847 } }
I20260929 10:55:10.781783 19919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d95325ad5934ff3b656fdd49694d615 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8d95325ad5934ff3b656fdd49694d615" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d95325ad5934ff3b656fdd49694d615" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 43847 } } }
I20260929 10:55:10.781879 19919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d95325ad5934ff3b656fdd49694d615 [sys.catalog]: This master's current role is: LEADER
I20260929 10:55:10.782101 19920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d95325ad5934ff3b656fdd49694d615 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d95325ad5934ff3b656fdd49694d615. Latest consensus state: current_term: 2 leader_uuid: "8d95325ad5934ff3b656fdd49694d615" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d95325ad5934ff3b656fdd49694d615" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 43847 } } }
I20260929 10:55:10.782202 19920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d95325ad5934ff3b656fdd49694d615 [sys.catalog]: This master's current role is: LEADER
I20260929 10:55:10.782325 19925 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:55:10.782838 19925 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:55:10.783015 19925 catalog_manager.cc:1259] Loaded cluster ID: 721d27f514884c39b199a7352fa6ca90
I20260929 10:55:10.783191 19925 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:55:10.784477 19925 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:55:10.784638 19925 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8d95325ad5934ff3b656fdd49694d615: Loaded TSK: 0
I20260929 10:55:10.784843 19925 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 10:55:10.854668 19916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:55:10.854914 19916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:55:10.854954 19916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:55:10.856138 19916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:55:10.856208 19916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:55:10.856253 19916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.193
I20260929 10:55:10.857746 19916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:45655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.193.keytab
--principal=oryx/127.10.238.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.193:40321
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.238.193
--webserver_port=45657
--tserver_master_addrs=127.10.238.254:43847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.193
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:55:10.858004 19916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:55:10.858234 19916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:55:10.864115 19943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:10.864137 19941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:55:10.864413 19916 server_base.cc:1034] running on GCE node
W20260929 10:55:10.864171 19940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:55:10.864711 19916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:55:10.864976 19916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:55:10.866130 19916 hybrid_clock.cc:648] HybridClock initialized: now 1790679310866108 us; error 37 us; skew 500 ppm
Sep 29 10:55:10 dist-test-slave-8nrt krb5kdc[19333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679310, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.238.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:55:10.869086 19916 init.cc:377] Logged in from keytab as oryx/127.10.238.193@KRBTEST.COM (short username oryx)
I20260929 10:55:10.869649 19916 webserver.cc:460] Webserver started at http://127.10.238.193:45657/ using document root <none> and password file <none>
I20260929 10:55:10.869927 19916 fs_manager.cc:359] Metadata directory not provided
I20260929 10:55:10.870025 19916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:55:10.871893 19916 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 10:55:10.872596 19950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:10.872807 19916 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 10:55:10.872910 19916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-0/wal
uuid: "081708d53308419db3732c5e5d5cefc9"
format_stamp: "Formatted at 2026-09-29 10:55:10 on dist-test-slave-8nrt"
server_key: "73d8f28ec27a2411a12ac6e68ae924dc"
server_key_iv: "f6b3a8d0f8ae4f4bdac0142a639dd582"
server_key_version: "encryptionkey@0"
I20260929 10:55:10.873413 19916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:55:10.892598 19916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:55:10.895429 19916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:55:10.895596 19916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:55:10.895884 19916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:55:10.896242 19916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:55:10.896291 19916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:10.896322 19916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:55:10.896340 19916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:10.903890 19916 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.193:40321
I20260929 10:55:10.903924 20063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.193:40321 every 8 connection(s)
I20260929 10:55:10.904383 19916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
Sep 29 10:55:10 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.193@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:10 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.193@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:55:10.913488 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 19916
I20260929 10:55:10.913805 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.194:43849
--local_ip_for_outbound_sockets=127.10.238.194
--tserver_master_addrs=127.10.238.254:43847
--webserver_port=37707
--webserver_interface=127.10.238.194
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab
--principal=kudu/127.10.238.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:45655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.10.238.194
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:55:10.913853 20066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:10.905204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.193:34703)
0929 10:55:10.905419 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:10.905463 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:10.905959 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:10.906685 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:10.906690 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:10.907292 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:10.907481 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:10.907486 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:10.908661 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:10.908662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:10.909576 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:10.909579 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:10.910207 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:10.910208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:10.910344 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:10.910911 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:10.910919 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:10.913578 (+ 2659us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":138,"threads_started":1}
W20260929 10:55:10.915424 20064 heartbeater.cc:644] Failed to heartbeat to 127.10.238.254:43847 (0 consecutive failures): Not authorized: Failed to ping master at 127.10.238.254:43847: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
W20260929 10:55:10.915753 20067 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:10.905312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:34703 (local address 127.10.238.254:43847)
0929 10:55:10.906118 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:10.906126 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:10.906162 (+ 36us) server_negotiation.cc:407] Connection header received
0929 10:55:10.906262 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:10.906266 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:10.906344 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:10.906606 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:10.907614 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:10.908542 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:10.909688 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:10.910105 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:10.910233 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:10.915566 (+ 5333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.238.193:34703: BlockingRecv error: recv got EOF from 127.10.238.193:34703 (error 108)
Metrics: {"server-negotiator.queue_time_us":759,"thread_start_us":96,"threads_started":1}
Sep 29 10:55:10 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.193@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:10 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.193@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:55:10.921772 20066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:10.915647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.193:46999)
0929 10:55:10.916032 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:10.916041 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:10.916113 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:10.916276 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:10.916277 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:10.916463 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:10.916521 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:10.916523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:10.917516 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:10.917517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:10.918327 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:10.918330 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:10.918899 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:10.918901 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:10.918990 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:10.919549 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:10.919555 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:10.921693 (+ 2138us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":371}
W20260929 10:55:10.922305 20067 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:10.915732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:46999 (local address 127.10.238.254:43847)
0929 10:55:10.915908 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:10.915910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:10.916099 (+ 189us) server_negotiation.cc:407] Connection header received
0929 10:55:10.916190 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:10.916191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:10.916209 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:10.916273 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:10.916601 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:10.917399 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:10.918429 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:10.918799 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:10.918878 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:10.921813 (+ 2935us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.238.193:46999: BlockingRecv error: recv got EOF from 127.10.238.193:46999 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":80,"threads_started":1}
Sep 29 10:55:10 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.193@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:10 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.193@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:55:10.927707 20066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:10.922222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.193:44397)
0929 10:55:10.922424 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:10.922436 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:10.922506 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:10.922681 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:10.922681 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:10.922831 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:10.922869 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:10.922871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:10.923733 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:10.923734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:10.924606 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:10.924609 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:10.925106 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:10.925107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:10.925162 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:10.925641 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:10.925647 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:10.927572 (+ 1925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184}
W20260929 10:55:10.927906 20064 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 10:55:10.927745 20069 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:10.922274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:44397 (local address 127.10.238.254:43847)
0929 10:55:10.922321 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:10.922323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:10.922480 (+ 157us) server_negotiation.cc:407] Connection header received
0929 10:55:10.922576 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:10.922577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:10.922587 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:10.922660 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:10.922944 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:10.923643 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:10.924706 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:10.925016 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:10.925094 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:10.927678 (+ 2584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.238.193:44397: BlockingRecv error: recv got EOF from 127.10.238.193:44397 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260929 10:55:11.000068 20068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:55:11.000304 20068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:55:11.000350 20068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:55:11.001523 20068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:55:11.001590 20068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:55:11.001636 20068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.194
I20260929 10:55:11.003180 20068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:45655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.194.keytab
--principal=oryx/127.10.238.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.194:43849
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.238.194
--webserver_port=37707
--tserver_master_addrs=127.10.238.254:43847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.194
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:55:11.003456 20068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:55:11.003690 20068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:55:11.008803 20075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:55:11.008973 20068 server_base.cc:1034] running on GCE node
W20260929 10:55:11.008831 20074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:11.008816 20077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:55:11.009210 20068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:55:11.009500 20068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:55:11.010667 20068 hybrid_clock.cc:648] HybridClock initialized: now 1790679311010657 us; error 40 us; skew 500 ppm
Sep 29 10:55:11 dist-test-slave-8nrt krb5kdc[19333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679311, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.238.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:55:11.013159 20068 init.cc:377] Logged in from keytab as oryx/127.10.238.194@KRBTEST.COM (short username oryx)
I20260929 10:55:11.013744 20068 webserver.cc:460] Webserver started at http://127.10.238.194:37707/ using document root <none> and password file <none>
I20260929 10:55:11.013983 20068 fs_manager.cc:359] Metadata directory not provided
I20260929 10:55:11.014063 20068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:55:11.015450 20068 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 10:55:11.016057 20084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:11.016230 20068 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 10:55:11.016333 20068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-1/wal
uuid: "a3ed66e3e60b4a92b331ce5a24645e07"
format_stamp: "Formatted at 2026-09-29 10:55:10 on dist-test-slave-8nrt"
server_key: "5da713172568d08abe42d0642be5a550"
server_key_iv: "ebd358e3a50bd57f081d7f87a14618f9"
server_key_version: "encryptionkey@0"
I20260929 10:55:11.016696 20068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:55:11.042742 20068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:55:11.044826 20068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:55:11.044996 20068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:55:11.045240 20068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:55:11.045615 20068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:55:11.045672 20068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:11.045714 20068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:55:11.045742 20068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:11.051906 20068 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.194:43849
I20260929 10:55:11.051951 20197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.194:43849 every 8 connection(s)
I20260929 10:55:11.052426 20068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
Sep 29 10:55:11 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.194@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:11 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.194@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:55:11.059821 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 20068
I20260929 10:55:11.060155 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.195:44805
--local_ip_for_outbound_sockets=127.10.238.195
--tserver_master_addrs=127.10.238.254:43847
--webserver_port=40837
--webserver_interface=127.10.238.195
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab
--principal=kudu/127.10.238.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:45655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.10.238.195
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:55:11.060516 20069 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:11.053799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:37565 (local address 127.10.238.254:43847)
0929 10:55:11.053866 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:11.053868 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:11.054065 (+ 197us) server_negotiation.cc:407] Connection header received
0929 10:55:11.054547 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:11.054548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:11.054560 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:11.054610 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:11.055513 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:11.056015 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:11.057007 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:11.057213 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:11.057334 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:11.060467 (+ 3133us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.238.194:37565: BlockingRecv error: recv got EOF from 127.10.238.194:37565 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 10:55:11.060590 20200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:11.053792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.194:37565)
0929 10:55:11.053945 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:11.053983 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:11.054468 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:11.054662 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:11.054666 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:11.055221 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:11.055439 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:11.055443 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:11.056095 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:11.056095 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:11.056908 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:11.056911 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:11.057358 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:11.057359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:11.057461 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:11.057954 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:11.057963 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:11.060357 (+ 2394us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":87,"threads_started":1}
W20260929 10:55:11.060863 20198 heartbeater.cc:644] Failed to heartbeat to 127.10.238.254:43847 (0 consecutive failures): Not authorized: Failed to ping master at 127.10.238.254:43847: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
Sep 29 10:55:11 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.194@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:11 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.194@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:55:11.065630 20200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:11.061072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.194:34773)
0929 10:55:11.061191 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:11.061201 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:11.061273 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:11.061449 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:11.061449 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:11.061581 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:11.061625 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:11.061626 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:11.062259 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:11.062260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:11.062768 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:11.062769 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:11.063124 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:11.063125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:11.063194 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:11.063649 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:11.063654 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:11.065545 (+ 1891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101}
W20260929 10:55:11.065680 20069 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:11.061125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:34773 (local address 127.10.238.254:43847)
0929 10:55:11.061182 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:11.061184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:11.061278 (+ 94us) server_negotiation.cc:407] Connection header received
0929 10:55:11.061365 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:11.061366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:11.061378 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:11.061446 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:11.061702 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:11.062179 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:11.062850 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:11.063042 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:11.063123 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:11.065614 (+ 2491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.238.194:34773: BlockingRecv error: recv got EOF from 127.10.238.194:34773 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 29 10:55:11 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.194@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:11 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.194@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:55:11.070319 20200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:11.065934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.194:52661)
0929 10:55:11.066011 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:11.066021 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:11.066066 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:11.066193 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:11.066193 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:11.066326 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:11.066352 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:11.066353 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:11.066967 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:11.066968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:11.067455 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:11.067456 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:11.067829 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:11.067830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:11.067882 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:11.068337 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:11.068341 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:11.070217 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":62}
W20260929 10:55:11.070338 20069 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:11.065974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:52661 (local address 127.10.238.254:43847)
0929 10:55:11.066009 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:11.066011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:11.066052 (+ 41us) server_negotiation.cc:407] Connection header received
0929 10:55:11.066124 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:11.066125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:11.066135 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:11.066188 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:11.066428 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:11.066886 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:11.067552 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:11.067742 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:11.067829 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:11.070288 (+ 2459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.238.194:52661: BlockingRecv error: recv got EOF from 127.10.238.194:52661 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260929 10:55:11.070456 20198 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 10:55:11.143587 20201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:55:11.143836 20201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:55:11.143875 20201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:55:11.145031 20201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:55:11.145099 20201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:55:11.145147 20201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.195
I20260929 10:55:11.146621 20201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:45655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/krb5kdc/oryx_127.10.238.195.keytab
--principal=oryx/127.10.238.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.195:44805
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.238.195
--webserver_port=40837
--tserver_master_addrs=127.10.238.254:43847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.195
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:55:11.146898 20201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:55:11.147130 20201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:55:11.152011 20209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:11.152004 20207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:55:11.152096 20201 server_base.cc:1034] running on GCE node
W20260929 10:55:11.152001 20206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:55:11.152429 20201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:55:11.152700 20201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:55:11.153863 20201 hybrid_clock.cc:648] HybridClock initialized: now 1790679311153859 us; error 36 us; skew 500 ppm
Sep 29 10:55:11 dist-test-slave-8nrt krb5kdc[19333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679311, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.238.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:55:11.156351 20201 init.cc:377] Logged in from keytab as oryx/127.10.238.195@KRBTEST.COM (short username oryx)
I20260929 10:55:11.156783 20201 webserver.cc:460] Webserver started at http://127.10.238.195:40837/ using document root <none> and password file <none>
I20260929 10:55:11.157008 20201 fs_manager.cc:359] Metadata directory not provided
I20260929 10:55:11.157088 20201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:55:11.158450 20201 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:55:11.159003 20216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:11.159165 20201 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:55:11.159260 20201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-2/wal
uuid: "d7f8ed4b94024663bdf2477437955545"
format_stamp: "Formatted at 2026-09-29 10:55:10 on dist-test-slave-8nrt"
server_key: "17df44e19bf3762b475a4623fef5c2e3"
server_key_iv: "ab27c11a0fc2d4c50d935654168b1751"
server_key_version: "encryptionkey@0"
I20260929 10:55:11.159605 20201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:55:11.174067 20201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:55:11.176276 20201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:55:11.176412 20201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:55:11.176594 20201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:55:11.176965 20201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:55:11.177011 20201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:11.177070 20201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:55:11.177099 20201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:11.183406 20201 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.195:44805
I20260929 10:55:11.183485 20329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.195:44805 every 8 connection(s)
I20260929 10:55:11.183851 20201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
I20260929 10:55:11.185222 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 20201
Sep 29 10:55:11 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.195@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:11 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.195@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
W20260929 10:55:11.192164 20069 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:11.184615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:57405 (local address 127.10.238.254:43847)
0929 10:55:11.184704 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:11.184706 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:11.184912 (+ 206us) server_negotiation.cc:407] Connection header received
0929 10:55:11.185501 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:11.185503 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:11.185514 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:11.185575 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:11.186609 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:11.187154 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:11.188364 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:11.188563 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:11.188631 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:11.192077 (+ 3446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.238.195:57405: BlockingRecv error: recv got EOF from 127.10.238.195:57405 (error 108)
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":55}
I20260929 10:55:11.192257 20332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:11.184582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.195:57405)
0929 10:55:11.184769 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:11.184812 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:11.185374 (+ 562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:11.185665 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:11.185669 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:11.186304 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:11.186505 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:11.186510 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:11.187267 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:11.187269 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:11.188262 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:11.188265 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:11.188667 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:11.188668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:11.188775 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:11.189408 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:11.189416 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:11.191990 (+ 2574us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":106,"threads_started":1}
W20260929 10:55:11.192471 20330 heartbeater.cc:644] Failed to heartbeat to 127.10.238.254:43847 (0 consecutive failures): Not authorized: Failed to ping master at 127.10.238.254:43847: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
I20260929 10:55:11.193795 20067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:11.185818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50870 (local address 127.10.238.254:43847)
0929 10:55:11.185869 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:11.185871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:11.185955 (+ 84us) server_negotiation.cc:407] Connection header received
0929 10:55:11.186025 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:11.186026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:11.186038 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:11.186096 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:11.186733 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:11.187488 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:11.188260 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:11.188505 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:11.188562 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:11.189927 (+ 1365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:11.189932 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:11.189933 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:11.189945 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:11.192064 (+ 2119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:11.192622 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:11.192625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:11.192626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:11.192800 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:11.193017 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:11.193018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:11.193019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:11.193302 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:11.193446 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:11.193557 (+ 111us) server_negotiation.cc:299] Negotiation successful
0929 10:55:11.193673 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":28}
Sep 29 10:55:11 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.195@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:11 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.195@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:55:11.197054 20332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:11.192687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.195:34153)
0929 10:55:11.192737 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:11.192747 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:11.192787 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:11.192942 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:11.192942 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:11.193072 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:11.193114 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:11.193115 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:11.193824 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:11.193825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:11.194565 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:11.194567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:11.194957 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:11.194958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:11.195008 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:11.195418 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:11.195423 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:11.197011 (+ 1588us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260929 10:55:11.197181 20069 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:11.192745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:34153 (local address 127.10.238.254:43847)
0929 10:55:11.192785 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:11.192787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:11.192796 (+ 9us) server_negotiation.cc:407] Connection header received
0929 10:55:11.192855 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:11.192857 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:11.192865 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:11.192936 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:11.193198 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:11.193747 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:11.194638 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:11.194892 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:11.194946 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:11.197109 (+ 2163us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.238.195:34153: BlockingRecv error: recv got EOF from 127.10.238.195:34153 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 29 10:55:11 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.195@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:11 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.195@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:55:11.200855 20332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:11.197346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.195:60973)
0929 10:55:11.197392 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:11.197402 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:11.197442 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:11.197540 (+ 98us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:11.197541 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:11.197640 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:11.197664 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:11.197665 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:11.198208 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:11.198209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:11.198666 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:11.198667 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:11.198967 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:11.198968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:11.199003 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:11.199414 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:11.199418 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:11.200814 (+ 1396us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260929 10:55:11.200982 20069 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:11.197411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:60973 (local address 127.10.238.254:43847)
0929 10:55:11.197443 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:11.197444 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:11.197452 (+ 8us) server_negotiation.cc:407] Connection header received
0929 10:55:11.197483 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:11.197484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:11.197491 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:11.197531 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:11.197718 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:11.198147 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:11.198735 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:11.198908 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:11.198954 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:11.200909 (+ 1955us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.238.195:60973: BlockingRecv error: recv got EOF from 127.10.238.195:60973 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260929 10:55:11.200994 20330 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 10:55:11 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.193@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:11 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.193@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:55:11.933333 20334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:11.928382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.193:53339)
0929 10:55:11.928527 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:11.928539 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:11.928601 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:11.928781 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:11.928782 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:11.928950 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:11.929004 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:11.929005 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:11.929705 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:11.929706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:11.930253 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:11.930255 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:11.930595 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:11.930596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:11.930640 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:11.931177 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:11.931182 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:11.933218 (+ 2036us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":89,"threads_started":1}
W20260929 10:55:11.933426 20335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:11.928444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:53339 (local address 127.10.238.254:43847)
0929 10:55:11.928586 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:11.928587 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:11.928598 (+ 11us) server_negotiation.cc:407] Connection header received
0929 10:55:11.928659 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:11.928659 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:11.928669 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:11.928736 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:11.929077 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:11.929639 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:11.930328 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:11.930528 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:11.930580 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:11.933354 (+ 2774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.238.193:53339: BlockingRecv error: recv got EOF from 127.10.238.193:53339 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":83,"threads_started":1}
Sep 29 10:55:12 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.194@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:12 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.194@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:55:12.075651 20336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:12.070884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.194:34031)
0929 10:55:12.071038 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:12.071049 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:12.071128 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:12.071254 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:12.071254 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:12.071431 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:12.071481 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:12.071483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:12.072151 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:12.072152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:12.072710 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:12.072711 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:12.073048 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:12.073049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:12.073087 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:12.073684 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:12.073690 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:12.075575 (+ 1885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":84,"threads_started":1}
W20260929 10:55:12.075758 20335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:12.070979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:34031 (local address 127.10.238.254:43847)
0929 10:55:12.071062 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:12.071064 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:12.071112 (+ 48us) server_negotiation.cc:407] Connection header received
0929 10:55:12.071173 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:12.071174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:12.071185 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:12.071234 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:12.071578 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:12.072082 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:12.072793 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:12.072980 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:12.073035 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:12.075665 (+ 2630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.238.194:34031: BlockingRecv error: recv got EOF from 127.10.238.194:34031 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 29 10:55:12 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.195@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:12 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.195@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:55:12.206331 20337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:12.201452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.195:39607)
0929 10:55:12.201631 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:12.201642 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:12.201711 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:12.201865 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:12.201866 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:12.202029 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:12.202079 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:12.202081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:12.202689 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:12.202689 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:12.203239 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:12.203240 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:12.203614 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:12.203615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:12.203657 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:12.204128 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:12.204133 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:12.206260 (+ 2127us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":110,"threads_started":1}
W20260929 10:55:12.206398 20335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:12.201545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:39607 (local address 127.10.238.254:43847)
0929 10:55:12.201622 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:12.201624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:12.201713 (+ 89us) server_negotiation.cc:407] Connection header received
0929 10:55:12.201766 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:12.201767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:12.201781 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:12.201840 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:12.202141 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:12.202624 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:12.203346 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:12.203539 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:12.203604 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:12.206351 (+ 2747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.238.195:39607: BlockingRecv error: recv got EOF from 127.10.238.195:39607 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 10:55:12 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.193@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:12 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.193@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:55:12.938465 20338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:12.933839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.193:59509)
0929 10:55:12.933980 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:12.933992 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:12.934053 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:12.934186 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:12.934186 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:12.934352 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:12.934405 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:12.934406 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:12.935053 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:12.935054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:12.935584 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:12.935585 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:12.935994 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:12.935996 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:12.936060 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:12.936638 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:12.936643 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:12.938407 (+ 1764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":88,"threads_started":1}
W20260929 10:55:12.938589 20339 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:12.933930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:59509 (local address 127.10.238.254:43847)
0929 10:55:12.934064 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:12.934066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:12.934074 (+ 8us) server_negotiation.cc:407] Connection header received
0929 10:55:12.934096 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:12.934097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:12.934107 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:12.934168 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:12.934472 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:12.934983 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:12.935677 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:12.935898 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:12.935967 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:12.938503 (+ 2536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.238.193:59509: BlockingRecv error: recv got EOF from 127.10.238.193:59509 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":72,"threads_started":1}
Sep 29 10:55:13 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.194@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:13 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.194@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:55:13.081323 20340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:13.076293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.194:38655)
0929 10:55:13.076482 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:13.076493 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:13.076560 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:13.076753 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:13.076754 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:13.076987 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:13.077063 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:13.077066 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:13.077896 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:13.077896 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:13.078487 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:13.078489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:13.078818 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:13.078818 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:13.078860 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:13.079372 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:13.079376 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:13.081198 (+ 1822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":110,"threads_started":1}
W20260929 10:55:13.081391 20339 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:13.076357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:38655 (local address 127.10.238.254:43847)
0929 10:55:13.076448 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:13.076450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:13.076592 (+ 142us) server_negotiation.cc:407] Connection header received
0929 10:55:13.076619 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:13.076620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:13.076631 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:13.076676 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:13.077162 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:13.077827 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:13.078564 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:13.078754 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:13.078808 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:13.081342 (+ 2534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.238.194:38655: BlockingRecv error: recv got EOF from 127.10.238.194:38655 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 29 10:55:13 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.195@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:13 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.195@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:55:13.211745 20341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:13.206893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.195:49513)
0929 10:55:13.207076 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:13.207091 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:13.207158 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:13.207386 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:13.207388 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:13.207597 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:13.207653 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:13.207656 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:13.208327 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:13.208329 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:13.208891 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:13.208893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:13.209302 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:13.209303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:13.209360 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:13.209807 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:13.209811 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:13.211685 (+ 1874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":61,"threads_started":1}
W20260929 10:55:13.211880 20339 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:13.207189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:49513 (local address 127.10.238.254:43847)
0929 10:55:13.207246 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:13.207249 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:13.207256 (+ 7us) server_negotiation.cc:407] Connection header received
0929 10:55:13.207278 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:13.207279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:13.207294 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:13.207375 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:13.207740 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:13.208228 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:13.208982 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:13.209176 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:13.209308 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:13.211769 (+ 2461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.238.195:49513: BlockingRecv error: recv got EOF from 127.10.238.195:49513 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 29 10:55:13 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.193@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:13 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.193@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:55:13.943593 20342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:13.938971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.193:41143)
0929 10:55:13.939145 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:13.939161 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:13.939218 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:13.939355 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:13.939356 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:13.939520 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:13.939588 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:13.939589 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:13.940218 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:13.940218 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:13.940755 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:13.940756 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:13.941081 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:13.941082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:13.941129 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:13.941715 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:13.941720 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:13.943522 (+ 1802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":91,"threads_started":1}
W20260929 10:55:13.943704 20343 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:13.939032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:41143 (local address 127.10.238.254:43847)
0929 10:55:13.939160 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:13.939162 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:13.939206 (+ 44us) server_negotiation.cc:407] Connection header received
0929 10:55:13.939273 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:13.939274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:13.939284 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:13.939332 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:13.939655 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:13.940145 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:13.940828 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:13.941018 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:13.941067 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:13.943624 (+ 2557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.238.193:41143: BlockingRecv error: recv got EOF from 127.10.238.193:41143 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":80,"threads_started":1}
Sep 29 10:55:14 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.194@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:14 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.194@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:55:14.086606 20344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:14.081929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.194:40367)
0929 10:55:14.082097 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:14.082112 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:14.082170 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:14.082324 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:14.082325 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:14.082504 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:14.082555 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:14.082557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:14.083202 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:14.083203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:14.083730 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:14.083731 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:14.084053 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:14.084054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:14.084094 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:14.084526 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:14.084530 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:14.086507 (+ 1977us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":114,"threads_started":1}
W20260929 10:55:14.086659 20343 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:14.082083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:40367 (local address 127.10.238.254:43847)
0929 10:55:14.082120 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:14.082122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:14.082156 (+ 34us) server_negotiation.cc:407] Connection header received
0929 10:55:14.082231 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:14.082232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:14.082244 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:14.082298 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:14.082613 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:14.083134 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:14.083804 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:14.083988 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:14.084040 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:14.086594 (+ 2554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.238.194:40367: BlockingRecv error: recv got EOF from 127.10.238.194:40367 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
Sep 29 10:55:14 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.195@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:14 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.195@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:55:14.217101 20345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:14.212178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.195:48811)
0929 10:55:14.212347 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:14.212359 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:14.212452 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:14.212603 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:14.212604 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:14.212788 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:14.212839 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:14.212841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:14.213703 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:14.213704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:14.214317 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:14.214319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:14.214670 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:14.214671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:14.214711 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:14.215257 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:14.215265 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:14.217013 (+ 1748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":94,"threads_started":1}
W20260929 10:55:14.217206 20343 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:14.212254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:48811 (local address 127.10.238.254:43847)
0929 10:55:14.212340 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:14.212342 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:14.212434 (+ 92us) server_negotiation.cc:407] Connection header received
0929 10:55:14.212530 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:14.212531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:14.212546 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:14.212598 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:14.212923 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:14.213627 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:14.214415 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:14.214607 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:14.214663 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:14.217128 (+ 2465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.238.195:48811: BlockingRecv error: recv got EOF from 127.10.238.195:48811 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 10:55:14 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.193@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:14 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.193@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:55:14.951898 20346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:14.944161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.193:45823)
0929 10:55:14.944331 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:14.944343 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:14.944405 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:14.944551 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:14.944552 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:14.944733 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:14.944785 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:14.944787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:14.945450 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:14.945450 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:14.945991 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:14.945992 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:14.946325 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:14.946326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:14.946369 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:14.949965 (+ 3596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:14.949970 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:14.951806 (+ 1836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":87,"threads_started":1}
W20260929 10:55:14.951978 20347 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:14.944268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:45823 (local address 127.10.238.254:43847)
0929 10:55:14.944382 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:14.944387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:14.944403 (+ 16us) server_negotiation.cc:407] Connection header received
0929 10:55:14.944466 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:14.944466 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:14.944477 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:14.944526 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:14.944858 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:14.945383 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:14.946069 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:14.946256 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:14.946316 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:14.951902 (+ 5586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.238.193:45823: BlockingRecv error: recv got EOF from 127.10.238.193:45823 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":61,"threads_started":1}
Sep 29 10:55:15 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.194@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:15 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.194@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:55:15.091774 20348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:15.087158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.194:46383)
0929 10:55:15.087311 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:15.087323 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:15.087390 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:15.087529 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:15.087529 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:15.087696 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:15.087747 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:15.087748 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:15.088370 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:15.088370 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:15.088890 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:15.088891 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:15.089300 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:15.089301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:15.089358 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:15.089883 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:15.089888 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:15.091714 (+ 1826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":117,"threads_started":1}
W20260929 10:55:15.091918 20347 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:15.087218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:46383 (local address 127.10.238.254:43847)
0929 10:55:15.087288 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:15.087291 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:15.087390 (+ 99us) server_negotiation.cc:407] Connection header received
0929 10:55:15.087451 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:15.087452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:15.087464 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:15.087505 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:15.087809 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:15.088305 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:15.088991 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:15.089200 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:15.089335 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:15.091826 (+ 2491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.238.194:46383: BlockingRecv error: recv got EOF from 127.10.238.194:46383 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 29 10:55:15 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.195@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:15 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.195@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:55:15.222932 20349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:15.217605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.195:33969)
0929 10:55:15.217747 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:15.217764 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:15.217846 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:15.217992 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:15.217993 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:15.218161 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:15.218210 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:15.218212 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:15.219097 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:15.219099 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:15.220021 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:15.220024 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:15.220386 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:15.220387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:15.220437 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:15.221024 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:15.221028 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:15.222870 (+ 1842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":85,"threads_started":1}
W20260929 10:55:15.223038 20347 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:15.217684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:33969 (local address 127.10.238.254:43847)
0929 10:55:15.217775 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:15.217777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:15.217830 (+ 53us) server_negotiation.cc:407] Connection header received
0929 10:55:15.217893 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:15.217894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:15.217907 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:15.217973 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:15.218278 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:15.219028 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:15.220105 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:15.220302 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:15.220385 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:15.222968 (+ 2583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.238.195:33969: BlockingRecv error: recv got EOF from 127.10.238.195:33969 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 29 10:55:15 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.193@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:15 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.193@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:55:15.957015 20350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:15.952405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.193:36177)
0929 10:55:15.952555 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:15.952565 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:15.952617 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:15.952766 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:15.952767 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:15.952929 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:15.952981 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:15.952983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:15.953652 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:15.953653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:15.954186 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:15.954188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:15.954547 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:15.954548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:15.954589 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:15.955063 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:15.955067 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:15.956906 (+ 1839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":91,"threads_started":1}
W20260929 10:55:15.957059 20351 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:15.952494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:36177 (local address 127.10.238.254:43847)
0929 10:55:15.952617 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:15.952620 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:15.952657 (+ 37us) server_negotiation.cc:407] Connection header received
0929 10:55:15.952679 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:15.952679 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:15.952690 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:15.952738 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:15.953049 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:15.953586 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:15.954278 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:15.954478 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:15.954533 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:15.956982 (+ 2449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.238.193:36177: BlockingRecv error: recv got EOF from 127.10.238.193:36177 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":54,"threads_started":1}
Sep 29 10:55:16 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.194@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:16 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.194@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:55:16.097007 20352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:16.092231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.194:56329)
0929 10:55:16.092383 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:16.092399 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:16.092486 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:16.092650 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:16.092650 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:16.092845 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:16.092895 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:16.092896 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:16.093653 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:16.093654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:16.094201 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:16.094202 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:16.094582 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:16.094583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:16.094645 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:16.095095 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:16.095100 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:16.096908 (+ 1808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":71,"threads_started":1}
W20260929 10:55:16.097056 20351 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:16.092396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:56329 (local address 127.10.238.254:43847)
0929 10:55:16.092461 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:16.092464 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:16.092473 (+ 9us) server_negotiation.cc:407] Connection header received
0929 10:55:16.092560 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:16.092562 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:16.092578 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:16.092641 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:16.092981 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:16.093567 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:16.094296 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:16.094493 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:16.094580 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:16.096991 (+ 2411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.238.194:56329: BlockingRecv error: recv got EOF from 127.10.238.194:56329 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 29 10:55:16 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.195@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:16 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.195@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:55:16.228101 20353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:16.223429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.195:43821)
0929 10:55:16.223585 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:16.223596 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:16.223657 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:16.223824 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:16.223824 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:16.223993 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:16.224046 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:16.224047 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:16.224759 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:16.224759 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:16.225313 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:16.225314 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:16.225705 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:16.225706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:16.225744 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:16.226187 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:16.226191 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:16.228043 (+ 1852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":94,"threads_started":1}
W20260929 10:55:16.228227 20351 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:16.223482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:43821 (local address 127.10.238.254:43847)
0929 10:55:16.223550 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:16.223552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:16.223696 (+ 144us) server_negotiation.cc:407] Connection header received
0929 10:55:16.223728 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:16.223729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:16.223746 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:16.223812 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:16.224131 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:16.224685 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:16.225403 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:16.225614 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:16.225685 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:16.228174 (+ 2489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.238.195:43821: BlockingRecv error: recv got EOF from 127.10.238.195:43821 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 29 10:55:16 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.193@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:16 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.193@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:55:16.962677 20354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:16.957612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.193:51711)
0929 10:55:16.957759 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:16.957771 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:16.957835 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:16.957979 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:16.957980 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:16.958137 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:16.958207 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:16.958208 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:16.958975 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:16.958976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:16.959602 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:16.959610 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:16.960037 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:16.960038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:16.960112 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:16.960576 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:16.960580 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:16.962601 (+ 2021us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":85,"threads_started":1}
W20260929 10:55:16.962845 20355 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:16.957706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:51711 (local address 127.10.238.254:43847)
0929 10:55:16.957844 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:16.957846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:16.957854 (+ 8us) server_negotiation.cc:407] Connection header received
0929 10:55:16.957879 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:16.957880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:16.957890 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:16.957957 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:16.958314 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:16.958898 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:16.959689 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:16.959957 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:16.960036 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:16.962701 (+ 2665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.238.193:51711: BlockingRecv error: recv got EOF from 127.10.238.193:51711 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":86,"threads_started":1}
Sep 29 10:55:17 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.194@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:17 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.194@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:55:17.102425 20356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:17.097529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.194:40243)
0929 10:55:17.097711 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:17.097724 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:17.097836 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:17.098015 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:17.098016 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:17.098200 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:17.098251 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:17.098252 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:17.098923 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:17.098924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:17.099458 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:17.099459 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:17.099824 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:17.099825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:17.099884 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:17.100364 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:17.100368 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:17.102327 (+ 1959us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":99,"threads_started":1}
W20260929 10:55:17.102465 20355 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:17.097637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:40243 (local address 127.10.238.254:43847)
0929 10:55:17.097715 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:17.097718 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:17.097802 (+ 84us) server_negotiation.cc:407] Connection header received
0929 10:55:17.097918 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:17.097920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:17.097938 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:17.097998 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:17.098344 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:17.098840 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:17.099550 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:17.099747 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:17.099823 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:17.102405 (+ 2582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.238.194:40243: BlockingRecv error: recv got EOF from 127.10.238.194:40243 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 29 10:55:17 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.195@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:17 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.195@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:55:17.233424 20357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:17.228766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.195:36671)
0929 10:55:17.228929 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:17.228940 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:17.229005 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:17.229129 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:17.229130 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:17.229314 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:17.229369 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:17.229370 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:17.229982 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:17.229983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:17.230521 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:17.230522 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:17.230847 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:17.230847 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:17.230889 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:17.231419 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:17.231424 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:17.233365 (+ 1941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":103,"threads_started":1}
W20260929 10:55:17.233533 20355 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:17.228824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:36671 (local address 127.10.238.254:43847)
0929 10:55:17.228895 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:17.228897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:17.229007 (+ 110us) server_negotiation.cc:407] Connection header received
0929 10:55:17.229047 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:17.229048 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:17.229061 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:17.229111 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:17.229431 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:17.229917 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:17.230592 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:17.230782 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:17.230833 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:17.233456 (+ 2623us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.238.195:36671: BlockingRecv error: recv got EOF from 127.10.238.195:36671 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 29 10:55:17 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.193@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:17 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.193@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:55:17.967855 20358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:17.963152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.193:42855)
0929 10:55:17.963298 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:17.963327 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:17.963414 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:17.963597 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:17.963598 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:17.963769 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:17.963823 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:17.963824 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:17.964482 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:17.964483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:17.965031 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:17.965033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:17.965454 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:17.965455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:17.965506 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:17.965963 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:17.965967 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:17.967755 (+ 1788us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":79,"threads_started":1}
W20260929 10:55:17.967917 20359 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:17.963348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:42855 (local address 127.10.238.254:43847)
0929 10:55:17.963472 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:17.963473 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:17.963481 (+ 8us) server_negotiation.cc:407] Connection header received
0929 10:55:17.963503 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:17.963503 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:17.963513 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:17.963575 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:17.963890 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:17.964387 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:17.965106 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:17.965371 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:17.965424 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:17.967844 (+ 2420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.238.193:42855: BlockingRecv error: recv got EOF from 127.10.238.193:42855 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":55,"threads_started":1}
Sep 29 10:55:18 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.194@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:18 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.194@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:55:18.107575 20360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:18.102957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.194:47063)
0929 10:55:18.103075 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:18.103088 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:18.103164 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:18.103303 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:18.103304 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:18.103472 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:18.103525 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:18.103527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:18.104176 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:18.104177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:18.104725 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:18.104727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:18.105075 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:18.105076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:18.105118 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:18.105609 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:18.105614 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:18.107516 (+ 1902us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":55,"threads_started":1}
W20260929 10:55:18.107683 20359 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:18.103042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:47063 (local address 127.10.238.254:43847)
0929 10:55:18.103132 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:18.103134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:18.103146 (+ 12us) server_negotiation.cc:407] Connection header received
0929 10:55:18.103222 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:18.103222 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:18.103234 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:18.103280 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:18.103590 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:18.104084 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:18.104818 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:18.105005 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:18.105064 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:18.107612 (+ 2548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.238.194:47063: BlockingRecv error: recv got EOF from 127.10.238.194:47063 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 29 10:55:18 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.195@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:18 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.195@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:55:18.238586 20361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:18.233927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.195:49519)
0929 10:55:18.234068 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:18.234080 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:18.234149 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:18.234278 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:18.234278 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:18.234455 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:18.234507 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:18.234508 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:18.235166 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:18.235167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:18.235769 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:18.235772 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:18.236094 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:18.236095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:18.236134 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:18.236592 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:18.236597 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:18.238488 (+ 1891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":88,"threads_started":1}
W20260929 10:55:18.238629 20359 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:18.233985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:49519 (local address 127.10.238.254:43847)
0929 10:55:18.234058 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:18.234060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:18.234150 (+ 90us) server_negotiation.cc:407] Connection header received
0929 10:55:18.234193 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:18.234194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:18.234206 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:18.234256 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:18.234571 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:18.235095 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:18.235847 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:18.236031 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:18.236081 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:18.238566 (+ 2485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.238.195:49519: BlockingRecv error: recv got EOF from 127.10.238.195:49519 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 29 10:55:18 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.193@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:18 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.193@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:55:18.973141 20362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:18.968392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.193:33165)
0929 10:55:18.968536 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:18.968555 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:18.968604 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:18.968788 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:18.968789 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:18.968976 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:18.969028 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:18.969029 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:18.969724 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:18.969725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:18.970244 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:18.970245 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:18.970567 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:18.970568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:18.970608 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:18.971141 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:18.971145 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:18.973034 (+ 1889us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":85,"threads_started":1}
W20260929 10:55:18.973189 20363 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:18.968514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:33165 (local address 127.10.238.254:43847)
0929 10:55:18.968638 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:18.968640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:18.968647 (+ 7us) server_negotiation.cc:407] Connection header received
0929 10:55:18.968671 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:18.968671 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:18.968682 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:18.968770 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:18.969095 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:18.969658 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:18.970312 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:18.970502 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:18.970554 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:18.973116 (+ 2562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.238.193:33165: BlockingRecv error: recv got EOF from 127.10.238.193:33165 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":70,"threads_started":1}
Sep 29 10:55:19 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.194@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:19 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.194@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:55:19.113077 20364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:19.108101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.194:42501)
0929 10:55:19.108281 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:19.108297 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:19.108392 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:19.108567 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:19.108568 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:19.108744 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:19.108796 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:19.108797 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:19.109691 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:19.109692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:19.110237 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:19.110239 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:19.110617 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:19.110618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:19.110681 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:19.111137 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:19.111141 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:19.112959 (+ 1818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":91,"threads_started":1}
W20260929 10:55:19.113142 20363 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:19.108186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:42501 (local address 127.10.238.254:43847)
0929 10:55:19.108269 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:19.108272 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:19.108372 (+ 100us) server_negotiation.cc:407] Connection header received
0929 10:55:19.108468 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:19.108469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:19.108484 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:19.108560 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:19.108898 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:19.109611 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:19.110329 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:19.110529 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:19.110615 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:19.113056 (+ 2441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.238.194:42501: BlockingRecv error: recv got EOF from 127.10.238.194:42501 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 10:55:19 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.195@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:19 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.195@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:55:19.243957 20365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:19.239058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.195:37023)
0929 10:55:19.239243 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:19.239256 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:19.239347 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:19.239497 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:19.239498 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:19.239682 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:19.239729 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:19.239731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:19.240418 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:19.240419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:19.240975 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:19.240976 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:19.241403 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:19.241404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:19.241470 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:19.241937 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:19.241942 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:19.243847 (+ 1905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":112,"threads_started":1}
W20260929 10:55:19.243996 20363 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:19.239134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:37023 (local address 127.10.238.254:43847)
0929 10:55:19.239230 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:19.239233 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:19.239328 (+ 95us) server_negotiation.cc:407] Connection header received
0929 10:55:19.239403 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:19.239404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:19.239419 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:19.239491 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:19.239808 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:19.240327 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:19.241061 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:19.241318 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:19.241400 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:19.243929 (+ 2529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.238.195:37023: BlockingRecv error: recv got EOF from 127.10.238.195:37023 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 29 10:55:19 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.193@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:19 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.193@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:55:19.982851 20366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:19.973710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.193:41703)
0929 10:55:19.973910 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:19.973923 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:19.974008 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:19.974214 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:19.974215 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:19.974383 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:19.974436 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:19.974437 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:19.975057 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:19.975057 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:19.975572 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:19.975573 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:19.975882 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:19.975882 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:19.975924 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:19.980887 (+ 4963us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:19.980893 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:19.982719 (+ 1826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":100,"threads_started":1}
W20260929 10:55:19.982877 20367 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:19.973923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:41703 (local address 127.10.238.254:43847)
0929 10:55:19.974078 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:19.974080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:19.974089 (+ 9us) server_negotiation.cc:407] Connection header received
0929 10:55:19.974121 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:19.974121 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:19.974132 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:19.974191 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:19.974501 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:19.974994 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:19.975636 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:19.975820 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:19.975868 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:19.982807 (+ 6939us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.238.193:41703: BlockingRecv error: recv got EOF from 127.10.238.193:41703 (error 108)
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":91,"threads_started":1}
Sep 29 10:55:20 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.194@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:20 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.194@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:55:20.118171 20368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:20.113515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.194:48841)
0929 10:55:20.113657 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:20.113666 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:20.113720 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:20.113853 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:20.113853 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:20.114013 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:20.114065 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:20.114066 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:20.114788 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:20.114789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:20.115318 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:20.115320 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:20.115686 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:20.115686 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:20.115725 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:20.116163 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:20.116168 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:20.118097 (+ 1929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":75,"threads_started":1}
W20260929 10:55:20.118320 20367 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:20.113574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:48841 (local address 127.10.238.254:43847)
0929 10:55:20.113621 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:20.113623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:20.113705 (+ 82us) server_negotiation.cc:407] Connection header received
0929 10:55:20.113768 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:20.113769 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:20.113780 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:20.113831 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:20.114126 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:20.114717 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:20.115391 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:20.115620 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:20.115673 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:20.118223 (+ 2550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.238.194:48841: BlockingRecv error: recv got EOF from 127.10.238.194:48841 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
Sep 29 10:55:20 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.195@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:20 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.195@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:55:20.249104 20369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:20.244542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.195:58607)
0929 10:55:20.244694 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:20.244707 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:20.244781 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:20.244932 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:20.244933 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:20.245096 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:20.245148 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:20.245150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:20.245857 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:20.245858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:20.246377 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:20.246378 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:20.246680 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:20.246681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:20.246718 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:20.247219 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:20.247225 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:20.249045 (+ 1820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":84,"threads_started":1}
W20260929 10:55:20.249317 20367 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:20.244645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:58607 (local address 127.10.238.254:43847)
0929 10:55:20.244697 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:20.244699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:20.244783 (+ 84us) server_negotiation.cc:407] Connection header received
0929 10:55:20.244851 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:20.244852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:20.244868 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:20.244915 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:20.245209 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:20.245783 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:20.246447 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:20.246620 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:20.246689 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:20.249210 (+ 2521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.238.195:58607: BlockingRecv error: recv got EOF from 127.10.238.195:58607 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 29 10:55:20 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.193@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:20 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.193@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:55:20.987998 20370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:20.983313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.193:51663)
0929 10:55:20.983461 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:20.983472 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:20.983522 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:20.983662 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:20.983663 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:20.983826 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:20.983884 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:20.983885 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:20.984629 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:20.984629 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:20.985154 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:20.985155 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:20.985544 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:20.985544 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:20.985588 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:20.986034 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:20.986038 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:20.987884 (+ 1846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":84,"threads_started":1}
W20260929 10:55:20.988106 20371 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:20.983377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:51663 (local address 127.10.238.254:43847)
0929 10:55:20.983523 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:20.983525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:20.983557 (+ 32us) server_negotiation.cc:407] Connection header received
0929 10:55:20.983579 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:20.983579 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:20.983590 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:20.983641 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:20.983948 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:20.984560 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:20.985228 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:20.985480 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:20.985533 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:20.988018 (+ 2485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.238.193:51663: BlockingRecv error: recv got EOF from 127.10.238.193:51663 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":94,"threads_started":1}
Sep 29 10:55:21 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.194@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:21 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.194@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:55:21.123261 20372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:21.118722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.194:60089)
0929 10:55:21.118860 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:21.118871 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:21.118928 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:21.119051 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:21.119052 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:21.119227 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:21.119280 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:21.119282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:21.119940 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:21.119941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:21.120496 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:21.120499 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:21.120814 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:21.120815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:21.120869 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:21.121372 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:21.121377 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:21.123202 (+ 1825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":80,"threads_started":1}
W20260929 10:55:21.123397 20371 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:21.118812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:60089 (local address 127.10.238.254:43847)
0929 10:55:21.118856 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:21.118858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:21.118913 (+ 55us) server_negotiation.cc:407] Connection header received
0929 10:55:21.118973 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:21.118974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:21.118984 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:21.119034 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:21.119343 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:21.119857 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:21.120573 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:21.120755 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:21.120799 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:21.123314 (+ 2515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.238.194:60089: BlockingRecv error: recv got EOF from 127.10.238.194:60089 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 29 10:55:21 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.195@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:21 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.195@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:55:21.254452 20373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:21.249589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.195:44941)
0929 10:55:21.249754 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:21.249769 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:21.249839 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:21.250028 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:21.250029 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:21.250216 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:21.250264 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:21.250266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:21.250941 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:21.250942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:21.251494 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:21.251496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:21.251854 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:21.251855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:21.251921 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:21.252367 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:21.252372 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:21.254383 (+ 2011us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":64,"threads_started":1}
W20260929 10:55:21.254580 20371 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:21.249711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:44941 (local address 127.10.238.254:43847)
0929 10:55:21.249783 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:21.249786 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:21.249820 (+ 34us) server_negotiation.cc:407] Connection header received
0929 10:55:21.249935 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:21.249936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:21.249959 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:21.250020 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:21.250346 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:21.250854 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:21.251582 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:21.251780 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:21.251855 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:21.254477 (+ 2622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.238.195:44941: BlockingRecv error: recv got EOF from 127.10.238.195:44941 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 29 10:55:21 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.193@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:21 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.193@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:55:21.993183 20374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:21.988476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.193:38583)
0929 10:55:21.988623 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:21.988633 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:21.988691 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:21.988851 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:21.988852 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:21.989016 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:21.989066 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:21.989067 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:21.989933 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:21.989934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:21.990510 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:21.990512 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:21.990834 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:21.990835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:21.990874 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:21.991376 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:21.991405 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:21.993113 (+ 1708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":84,"threads_started":1}
W20260929 10:55:21.993307 20375 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:21.988570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:38583 (local address 127.10.238.254:43847)
0929 10:55:21.988701 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:21.988703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:21.988711 (+ 8us) server_negotiation.cc:407] Connection header received
0929 10:55:21.988734 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:21.988735 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:21.988766 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:21.988827 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:21.989133 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:21.989875 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:21.990576 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:21.990764 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:21.990813 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:21.993205 (+ 2392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.238.193:38583: BlockingRecv error: recv got EOF from 127.10.238.193:38583 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":77,"threads_started":1}
Sep 29 10:55:22 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.194@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:22 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.194@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:55:22.128291 20376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:22.123827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.194:55713)
0929 10:55:22.123999 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:22.124010 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:22.124068 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:22.124199 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:22.124199 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:22.124374 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:22.124424 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:22.124426 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:22.125044 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:22.125045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:22.125630 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:22.125632 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:22.125950 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:22.125951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:22.125993 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:22.126444 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:22.126448 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:22.128207 (+ 1759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":110,"threads_started":1}
W20260929 10:55:22.128360 20375 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:22.123940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:55713 (local address 127.10.238.254:43847)
0929 10:55:22.124005 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:22.124007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:22.124064 (+ 57us) server_negotiation.cc:407] Connection header received
0929 10:55:22.124114 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:22.124114 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:22.124125 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:22.124180 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:22.124483 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:22.124967 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:22.125704 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:22.125883 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:22.125933 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:22.128298 (+ 2365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.238.194:55713: BlockingRecv error: recv got EOF from 127.10.238.194:55713 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 29 10:55:22 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.195@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:22 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.195@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:55:22.259496 20377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:22.255006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.195:39759)
0929 10:55:22.255186 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:22.255197 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:22.255260 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:22.255388 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:22.255389 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:22.255570 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:22.255617 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:22.255619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:22.256277 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:22.256278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:22.256833 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:22.256834 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:22.257148 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:22.257149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:22.257189 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:22.257700 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:22.257705 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:22.259430 (+ 1725us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":107,"threads_started":1}
W20260929 10:55:22.259588 20375 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:22.255069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:39759 (local address 127.10.238.254:43847)
0929 10:55:22.255155 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:22.255158 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:22.255270 (+ 112us) server_negotiation.cc:407] Connection header received
0929 10:55:22.255301 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:22.255301 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:22.255315 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:22.255373 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:22.255707 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:22.256204 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:22.256908 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:22.257085 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:22.257134 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:22.259521 (+ 2387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.238.195:39759: BlockingRecv error: recv got EOF from 127.10.238.195:39759 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 29 10:55:22 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.193@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:22 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.193@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:55:22.998474 20378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:22.993811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.193:47083)
0929 10:55:22.993933 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:22.993949 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:22.994010 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:22.994142 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:22.994143 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:22.994321 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:22.994368 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:22.994369 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:22.995108 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:22.995109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:22.995647 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:22.995649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:22.996007 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:22.996008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:22.996052 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:22.996595 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:22.996601 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:22.998414 (+ 1813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":70,"threads_started":1}
W20260929 10:55:22.998636 20379 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:22.993881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:47083 (local address 127.10.238.254:43847)
0929 10:55:22.994004 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:22.994007 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:22.994017 (+ 10us) server_negotiation.cc:407] Connection header received
0929 10:55:22.994066 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:22.994067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:22.994078 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:22.994138 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:22.994428 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:22.994948 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:22.995742 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:22.995936 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:22.995994 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:22.998538 (+ 2544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.238.193:47083: BlockingRecv error: recv got EOF from 127.10.238.193:47083 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":76,"threads_started":1}
Sep 29 10:55:23 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.194@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:23 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.194@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:55:23.133738 20380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:23.128884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.194:42471)
0929 10:55:23.129056 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:23.129076 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:23.129141 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:23.129351 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:23.129352 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:23.129523 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:23.129572 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:23.129582 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:23.130225 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:23.130226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:23.130771 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:23.130773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:23.131094 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:23.131096 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:23.131152 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:23.131637 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:23.131641 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:23.133628 (+ 1987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":75,"threads_started":1}
W20260929 10:55:23.133809 20379 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:23.129008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:42471 (local address 127.10.238.254:43847)
0929 10:55:23.129084 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:23.129085 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:23.129125 (+ 40us) server_negotiation.cc:407] Connection header received
0929 10:55:23.129212 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:23.129212 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:23.129223 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:23.129350 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:23.129644 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:23.130126 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:23.130845 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:23.131025 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:23.131074 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:23.133717 (+ 2643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.238.194:42471: BlockingRecv error: recv got EOF from 127.10.238.194:42471 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 10:55:23 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.195@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:23 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.195@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:55:23.265507 20381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:23.260105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.195:49027)
0929 10:55:23.260266 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:23.260279 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:23.260362 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:23.260518 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:23.260518 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:23.260748 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:23.260804 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:23.260806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:23.261527 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:23.261529 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:23.262277 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:23.262280 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:23.262658 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:23.262659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:23.262718 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:23.263213 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:23.263218 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:23.265448 (+ 2230us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":101,"threads_started":1}
W20260929 10:55:23.265623 20379 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:23.260216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:49027 (local address 127.10.238.254:43847)
0929 10:55:23.260271 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:23.260273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:23.260341 (+ 68us) server_negotiation.cc:407] Connection header received
0929 10:55:23.260431 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:23.260432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:23.260444 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:23.260486 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:23.260882 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:23.261437 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:23.262380 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:23.262562 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:23.262613 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:23.265545 (+ 2932us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.238.195:49027: BlockingRecv error: recv got EOF from 127.10.238.195:49027 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 29 10:55:23 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.193@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:23 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.193@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:55:24.003638 20382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:23.998936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.193:44495)
0929 10:55:23.999101 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:23.999111 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:23.999171 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:23.999305 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:23.999306 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:23.999474 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:23.999525 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:23.999527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:24.000160 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:24.000161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:24.000689 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:24.000690 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:24.001059 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:24.001061 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:24.001117 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:24.001762 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:24.001767 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:24.003577 (+ 1810us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":103,"threads_started":1}
W20260929 10:55:24.003764 20383 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:23.998979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:44495 (local address 127.10.238.254:43847)
0929 10:55:23.999115 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:23.999117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:23.999157 (+ 40us) server_negotiation.cc:407] Connection header received
0929 10:55:23.999226 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:23.999226 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:23.999237 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:23.999288 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:23.999592 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:24.000086 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:24.000775 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:24.000962 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:24.001013 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:24.003678 (+ 2665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.238.193:44495: BlockingRecv error: recv got EOF from 127.10.238.193:44495 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":76,"threads_started":1}
Sep 29 10:55:24 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.194@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:24 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.194@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:55:24.139145 20384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:24.134254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.194:56241)
0929 10:55:24.134418 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:24.134432 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:24.134505 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:24.134666 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:24.134667 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:24.134865 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:24.134913 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:24.134914 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:24.135702 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:24.135704 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:24.136418 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:24.136421 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:24.136758 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:24.136759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:24.136801 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:24.137351 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:24.137364 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:24.139086 (+ 1722us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":110,"threads_started":1}
W20260929 10:55:24.139257 20383 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:24.134394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:56241 (local address 127.10.238.254:43847)
0929 10:55:24.134452 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:24.134454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:24.134486 (+ 32us) server_negotiation.cc:407] Connection header received
0929 10:55:24.134567 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:24.134568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:24.134587 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:24.134646 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:24.134997 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:24.135594 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:24.136497 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:24.136686 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:24.136744 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:24.139184 (+ 2440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.238.194:56241: BlockingRecv error: recv got EOF from 127.10.238.194:56241 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 29 10:55:24 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.195@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:24 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.195@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:55:24.271347 20385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:24.266044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.195:41069)
0929 10:55:24.266245 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:24.266257 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:24.266353 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:24.266540 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:24.266541 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:24.266782 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:24.266856 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:24.266859 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:24.267595 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:24.267596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:24.268254 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:24.268256 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:24.268636 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:24.268638 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:24.268700 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:24.269344 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:24.269350 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:24.271262 (+ 1912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":99,"threads_started":1}
W20260929 10:55:24.271407 20383 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:24.266135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:41069 (local address 127.10.238.254:43847)
0929 10:55:24.266229 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:24.266231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:24.266354 (+ 123us) server_negotiation.cc:407] Connection header received
0929 10:55:24.266419 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:24.266421 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:24.266440 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:24.266523 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:24.266954 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:24.267517 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:24.268334 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:24.268533 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:24.268593 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:24.271357 (+ 2764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.238.195:41069: BlockingRecv error: recv got EOF from 127.10.238.195:41069 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 29 10:55:25 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.193@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:25 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.193@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:55:25.016016 20386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:25.004158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.193:49483)
0929 10:55:25.004332 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:25.004343 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:25.004403 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:25.004535 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:25.004536 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:25.004704 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:25.004754 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:25.004756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:25.005579 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:25.005580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:25.006221 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:25.006224 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:25.006565 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:25.006566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:25.006606 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:25.013883 (+ 7277us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:25.013906 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:25.015894 (+ 1988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":101,"threads_started":1}
W20260929 10:55:25.016115 20387 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:25.004257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:49483 (local address 127.10.238.254:43847)
0929 10:55:25.004384 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:25.004386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:25.004396 (+ 10us) server_negotiation.cc:407] Connection header received
0929 10:55:25.004449 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:25.004450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:25.004460 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:25.004520 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:25.004821 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:25.005473 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:25.006292 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:25.006498 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:25.006554 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:25.016014 (+ 9460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.238.193:49483: BlockingRecv error: recv got EOF from 127.10.238.193:49483 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":54,"threads_started":1}
Sep 29 10:55:25 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.194@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:25 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.194@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:55:25.144408 20388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:25.139650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.194:44029)
0929 10:55:25.139816 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:25.139829 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:25.139918 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:25.140089 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:25.140090 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:25.140279 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:25.140338 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:25.140340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:25.141017 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:25.141019 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:25.141643 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:25.141645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:25.142021 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:25.142022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:25.142085 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:25.142558 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:25.142563 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:25.144278 (+ 1715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":94,"threads_started":1}
W20260929 10:55:25.144465 20387 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:25.139815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:44029 (local address 127.10.238.254:43847)
0929 10:55:25.139892 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:25.139893 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:25.139903 (+ 10us) server_negotiation.cc:407] Connection header received
0929 10:55:25.139990 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:25.139991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:25.140003 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:25.140083 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:25.140424 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:25.140930 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:25.141728 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:25.141937 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:25.142011 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:25.144393 (+ 2382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.238.194:44029: BlockingRecv error: recv got EOF from 127.10.238.194:44029 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 29 10:55:25 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.195@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:25 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.195@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:55:25.276474 20389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:25.271844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.195:59307)
0929 10:55:25.271997 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:25.272007 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:25.272065 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:25.272200 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:25.272201 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:25.272373 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:25.272423 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:25.272424 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:25.273067 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:25.273068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:25.273658 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:25.273660 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:25.274002 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:25.274002 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:25.274042 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:25.274467 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:25.274471 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:25.276413 (+ 1942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":85,"threads_started":1}
W20260929 10:55:25.276605 20387 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:25.271920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:59307 (local address 127.10.238.254:43847)
0929 10:55:25.271989 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:25.271991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:25.272053 (+ 62us) server_negotiation.cc:407] Connection header received
0929 10:55:25.272123 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:25.272124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:25.272137 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:25.272181 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:25.272488 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:25.272991 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:25.273728 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:25.273928 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:25.273988 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:25.276502 (+ 2514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.238.195:59307: BlockingRecv error: recv got EOF from 127.10.238.195:59307 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 29 10:55:26 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.193@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:26 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.193@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:55:26.021006 20390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:26.016585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.193:36769)
0929 10:55:26.016738 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:26.016751 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:26.016820 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:26.016958 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:26.016958 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:26.017120 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:26.017172 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:26.017173 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:26.017846 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:26.017847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:26.018365 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:26.018366 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:26.018688 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:26.018689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:26.018727 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:26.019191 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:26.019198 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:26.020948 (+ 1750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":71,"threads_started":1}
W20260929 10:55:26.021124 20391 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:26.016646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:36769 (local address 127.10.238.254:43847)
0929 10:55:26.016795 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:26.016797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:26.016806 (+ 9us) server_negotiation.cc:407] Connection header received
0929 10:55:26.016877 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:26.016877 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:26.016887 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:26.016937 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:26.017234 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:26.017768 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:26.018437 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:26.018625 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:26.018673 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:26.021042 (+ 2369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.238.193:36769: BlockingRecv error: recv got EOF from 127.10.238.193:36769 (error 108)
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":83,"threads_started":1}
Sep 29 10:55:26 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.194@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:55:26 dist-test-slave-8nrt krb5kdc[19333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.238.194@KRBTEST.COM for oryx/127.10.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:55:26.149946 20392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:26.145159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:43847 (local address 127.10.238.194:34523)
0929 10:55:26.145321 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:26.145333 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:26.145399 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:26.145532 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:26.145533 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:26.145696 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:26.145749 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:26.145750 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:26.146415 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:26.146416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:26.146992 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:26.146995 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:26.147446 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:26.147447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:26.147518 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:26.148054 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:26.148059 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:26.149886 (+ 1827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.238.254:43847: Server oryx/127.10.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":99,"threads_started":1}
W20260929 10:55:26.150089 20391 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:26.145244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:34523 (local address 127.10.238.254:43847)
0929 10:55:26.145342 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:26.145344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:26.145386 (+ 42us) server_negotiation.cc:407] Connection header received
0929 10:55:26.145445 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:26.145445 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:26.145456 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:26.145507 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:26.145826 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:26.146318 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:26.147146 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:26.147362 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:26.147445 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:26.150006 (+ 2561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.238.194:34523: BlockingRecv error: recv got EOF from 127.10.238.194:34523 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 10:55:26.188000 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 19916
I20260929 10:55:26.192783 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 20068
I20260929 10:55:26.197186 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 20201
I20260929 10:55:26.201448 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 19846
2026-09-29T10:55:26Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17506 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T10:55:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:55:26Z Disabled control of system clock
I20260929 10:55:26.229357 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:45401
--webserver_interface=127.10.238.254
--webserver_port=0
--builtin_ntp_servers=127.10.238.212:38631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:45401
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 10:55:26.310026 20396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:55:26.310248 20396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:55:26.310269 20396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:55:26.311435 20396 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:55:26.311478 20396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:55:26.311491 20396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:55:26.311503 20396 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:55:26.311515 20396 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:55:26.312839 20396 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:38631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:45401
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:45401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:55:26.313046 20396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:55:26.313407 20396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:55:26.318682 20404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:55:26.318713 20396 server_base.cc:1034] running on GCE node
W20260929 10:55:26.318688 20402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:26.318682 20401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:55:26.319181 20396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:55:26.319517 20396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:55:26.320698 20396 hybrid_clock.cc:648] HybridClock initialized: now 1790679326320670 us; error 57 us; skew 500 ppm
I20260929 10:55:26.322016 20396 webserver.cc:460] Webserver started at http://127.10.238.254:43573/ using document root <none> and password file <none>
I20260929 10:55:26.322280 20396 fs_manager.cc:359] Metadata directory not provided
I20260929 10:55:26.322352 20396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:55:26.322505 20396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:55:26.323618 20396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/master-0/data/instance:
uuid: "bad8cdb465004d7cace3c40c6119979d"
format_stamp: "Formatted at 2026-09-29 10:55:26 on dist-test-slave-8nrt"
server_key: "60ca6208e0cbbeb05e32083a4276337b"
server_key_iv: "b2e86d8a111be18dc97aeb691d5a3cff"
server_key_version: "encryptionkey@0"
I20260929 10:55:26.324004 20396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/master-0/wal/instance:
uuid: "bad8cdb465004d7cace3c40c6119979d"
format_stamp: "Formatted at 2026-09-29 10:55:26 on dist-test-slave-8nrt"
server_key: "60ca6208e0cbbeb05e32083a4276337b"
server_key_iv: "b2e86d8a111be18dc97aeb691d5a3cff"
server_key_version: "encryptionkey@0"
I20260929 10:55:26.325372 20396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 10:55:26.326146 20410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:26.326309 20396 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:55:26.326418 20396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "bad8cdb465004d7cace3c40c6119979d"
format_stamp: "Formatted at 2026-09-29 10:55:26 on dist-test-slave-8nrt"
server_key: "60ca6208e0cbbeb05e32083a4276337b"
server_key_iv: "b2e86d8a111be18dc97aeb691d5a3cff"
server_key_version: "encryptionkey@0"
I20260929 10:55:26.326498 20396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:55:26.343364 20396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:55:26.343719 20396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:55:26.343883 20396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:55:26.348150 20396 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.254:45401
I20260929 10:55:26.348186 20462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.254:45401 every 8 connection(s)
I20260929 10:55:26.348567 20396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:55:26.349231 20463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:55:26.351894 20463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bad8cdb465004d7cace3c40c6119979d: Bootstrap starting.
I20260929 10:55:26.352511 20463 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bad8cdb465004d7cace3c40c6119979d: Neither blocks nor log segments found. Creating new log.
I20260929 10:55:26.352824 20463 log.cc:824] T 00000000000000000000000000000000 P bad8cdb465004d7cace3c40c6119979d: Log is configured to *not* fsync() on all Append() calls
I20260929 10:55:26.353565 20463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bad8cdb465004d7cace3c40c6119979d: No bootstrap required, opened a new log
I20260929 10:55:26.353843 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 20396
I20260929 10:55:26.353961 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/master-0/wal/instance
I20260929 10:55:26.354175 11195 external_mini_cluster.cc:1428] Setting key 4ae04822cae1949a74182210685c1951
I20260929 10:55:26.355870 20463 raft_consensus.cc:348] T 00000000000000000000000000000000 P bad8cdb465004d7cace3c40c6119979d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bad8cdb465004d7cace3c40c6119979d" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 45401 } }
I20260929 10:55:26.356045 20463 raft_consensus.cc:374] T 00000000000000000000000000000000 P bad8cdb465004d7cace3c40c6119979d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:55:26.356081 20463 raft_consensus.cc:729] T 00000000000000000000000000000000 P bad8cdb465004d7cace3c40c6119979d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bad8cdb465004d7cace3c40c6119979d, State: Initialized, Role: FOLLOWER
I20260929 10:55:26.356237 20463 consensus_queue.cc:260] T 00000000000000000000000000000000 P bad8cdb465004d7cace3c40c6119979d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bad8cdb465004d7cace3c40c6119979d" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 45401 } }
I20260929 10:55:26.356333 20463 raft_consensus.cc:388] T 00000000000000000000000000000000 P bad8cdb465004d7cace3c40c6119979d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:55:26.356365 20463 raft_consensus.cc:482] T 00000000000000000000000000000000 P bad8cdb465004d7cace3c40c6119979d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:55:26.356431 20463 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bad8cdb465004d7cace3c40c6119979d [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:55:26.357159 20463 raft_consensus.cc:504] T 00000000000000000000000000000000 P bad8cdb465004d7cace3c40c6119979d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bad8cdb465004d7cace3c40c6119979d" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 45401 } }
I20260929 10:55:26.357303 20463 leader_election.cc:302] T 00000000000000000000000000000000 P bad8cdb465004d7cace3c40c6119979d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bad8cdb465004d7cace3c40c6119979d; no voters:
I20260929 10:55:26.357456 20463 leader_election.cc:288] T 00000000000000000000000000000000 P bad8cdb465004d7cace3c40c6119979d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:55:26.357522 20468 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bad8cdb465004d7cace3c40c6119979d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:55:26.357672 20468 raft_consensus.cc:686] T 00000000000000000000000000000000 P bad8cdb465004d7cace3c40c6119979d [term 1 LEADER]: Becoming Leader. State: Replica: bad8cdb465004d7cace3c40c6119979d, State: Running, Role: LEADER
I20260929 10:55:26.357692 20463 sys_catalog.cc:564] T 00000000000000000000000000000000 P bad8cdb465004d7cace3c40c6119979d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:55:26.357816 20468 consensus_queue.cc:237] T 00000000000000000000000000000000 P bad8cdb465004d7cace3c40c6119979d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bad8cdb465004d7cace3c40c6119979d" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 45401 } }
I20260929 10:55:26.358136 20470 sys_catalog.cc:455] T 00000000000000000000000000000000 P bad8cdb465004d7cace3c40c6119979d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bad8cdb465004d7cace3c40c6119979d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bad8cdb465004d7cace3c40c6119979d" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 45401 } } }
I20260929 10:55:26.358204 20470 sys_catalog.cc:458] T 00000000000000000000000000000000 P bad8cdb465004d7cace3c40c6119979d [sys.catalog]: This master's current role is: LEADER
I20260929 10:55:26.358805 20475 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:55:26.359087 20469 sys_catalog.cc:455] T 00000000000000000000000000000000 P bad8cdb465004d7cace3c40c6119979d [sys.catalog]: SysCatalogTable state changed. Reason: New leader bad8cdb465004d7cace3c40c6119979d. Latest consensus state: current_term: 1 leader_uuid: "bad8cdb465004d7cace3c40c6119979d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bad8cdb465004d7cace3c40c6119979d" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 45401 } } }
I20260929 10:55:26.359186 20469 sys_catalog.cc:458] T 00000000000000000000000000000000 P bad8cdb465004d7cace3c40c6119979d [sys.catalog]: This master's current role is: LEADER
I20260929 10:55:26.359272 20475 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:55:26.361205 20475 catalog_manager.cc:1347] Generated new cluster ID: 5949464658a8417fb2a3c4b91de0147e
I20260929 10:55:26.361276 20475 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:55:26.361795 20467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:26.354754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38506 (local address 127.10.238.254:45401)
0929 10:55:26.355026 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:26.355035 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:26.355068 (+ 33us) server_negotiation.cc:407] Connection header received
0929 10:55:26.355821 (+ 753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:26.355831 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:26.355913 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:26.356073 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:26.356285 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:26.358081 (+ 1796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:26.359333 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:26.359985 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:26.360109 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:26.360847 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:26.360851 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:26.360851 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:55:26.361082 (+ 231us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:26.361123 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:55:26.361137 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:26.361463 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:26.361495 (+ 32us) server_negotiation.cc:299] Negotiation successful
0929 10:55:26.361570 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":117,"threads_started":1}
I20260929 10:55:26.374204 20475 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:55:26.374948 20475 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:55:26.380944 20475 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bad8cdb465004d7cace3c40c6119979d: Generated new TSK 0
I20260929 10:55:26.381134 20475 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:55:26.385509 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.193:0
--local_ip_for_outbound_sockets=127.10.238.193
--webserver_interface=127.10.238.193
--webserver_port=0
--tserver_master_addrs=127.10.238.254:45401
--builtin_ntp_servers=127.10.238.212:38631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 10:55:26.467222 20487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:55:26.467464 20487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:55:26.467499 20487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:55:26.468647 20487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:55:26.468714 20487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:55:26.468755 20487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.193
I20260929 10:55:26.470202 20487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:38631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.238.193
--webserver_port=0
--tserver_master_addrs=127.10.238.254:45401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.193
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:55:26.470465 20487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:55:26.470757 20487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:55:26.476625 20493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:55:26.476796 20487 server_base.cc:1034] running on GCE node
W20260929 10:55:26.476590 20495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:26.476590 20492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:55:26.477041 20487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:55:26.477366 20487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:55:26.478521 20487 hybrid_clock.cc:648] HybridClock initialized: now 1790679326478508 us; error 33 us; skew 500 ppm
I20260929 10:55:26.479851 20487 webserver.cc:460] Webserver started at http://127.10.238.193:43233/ using document root <none> and password file <none>
I20260929 10:55:26.480093 20487 fs_manager.cc:359] Metadata directory not provided
I20260929 10:55:26.480160 20487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:55:26.480299 20487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:55:26.481477 20487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-0/data/instance:
uuid: "6854bba2c4fb40389402f9cb47cd442a"
format_stamp: "Formatted at 2026-09-29 10:55:26 on dist-test-slave-8nrt"
server_key: "be9cafc224dfaed9af7f5934de27675c"
server_key_iv: "cf47975808e3b95b39d6e228bd92e3ce"
server_key_version: "encryptionkey@0"
I20260929 10:55:26.481841 20487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-0/wal/instance:
uuid: "6854bba2c4fb40389402f9cb47cd442a"
format_stamp: "Formatted at 2026-09-29 10:55:26 on dist-test-slave-8nrt"
server_key: "be9cafc224dfaed9af7f5934de27675c"
server_key_iv: "cf47975808e3b95b39d6e228bd92e3ce"
server_key_version: "encryptionkey@0"
I20260929 10:55:26.483194 20487 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:55:26.483989 20501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:26.484169 20487 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:55:26.484265 20487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-0/wal
uuid: "6854bba2c4fb40389402f9cb47cd442a"
format_stamp: "Formatted at 2026-09-29 10:55:26 on dist-test-slave-8nrt"
server_key: "be9cafc224dfaed9af7f5934de27675c"
server_key_iv: "cf47975808e3b95b39d6e228bd92e3ce"
server_key_version: "encryptionkey@0"
I20260929 10:55:26.484360 20487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:55:26.505405 20487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:55:26.505741 20487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:55:26.505905 20487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:55:26.506136 20487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:55:26.506512 20487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:55:26.506567 20487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:26.506601 20487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:55:26.506628 20487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:26.512717 20487 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.193:35777
I20260929 10:55:26.512790 20614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.193:35777 every 8 connection(s)
I20260929 10:55:26.513166 20487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
I20260929 10:55:26.518411 20467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:26.513990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:37641 (local address 127.10.238.254:45401)
0929 10:55:26.514065 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:26.514067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:26.514325 (+ 258us) server_negotiation.cc:407] Connection header received
0929 10:55:26.514964 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:26.514965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:26.514975 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:26.515027 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:26.515655 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:26.516141 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:26.517088 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:26.517281 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:26.517336 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:26.518167 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:26.518169 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:26.518169 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:55:26.518173 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:26.518188 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:55:26.518191 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:26.518238 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:26.518332 (+ 94us) server_negotiation.cc:299] Negotiation successful
0929 10:55:26.518353 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 10:55:26.518615 20617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:26.513991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:45401 (local address 127.10.238.193:37641)
0929 10:55:26.514190 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:26.514225 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:26.514870 (+ 645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:26.515086 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:26.515089 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:26.515097 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:26.515587 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:26.515591 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:26.516220 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:26.516220 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:26.517015 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:26.517017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:26.517361 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:26.517362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:26.517484 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:26.518025 (+ 541us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:55:26.518026 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:26.518055 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:55:26.518078 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:55:26.518253 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:55:26.518257 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:55:26.518261 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 10:55:26.518315 (+ 54us) client_negotiation.cc:240] Negotiation successful
0929 10:55:26.518399 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":94,"threads_started":1}
I20260929 10:55:26.519155 20615 heartbeater.cc:347] Connected to a master server at 127.10.238.254:45401
I20260929 10:55:26.519273 20615 heartbeater.cc:464] Registering TS with master...
I20260929 10:55:26.519459 20615 heartbeater.cc:511] Master 127.10.238.254:45401 requested a full tablet report, sending...
I20260929 10:55:26.519986 20427 ts_manager.cc:194] Registered new tserver with Master: 6854bba2c4fb40389402f9cb47cd442a (127.10.238.193:35777)
I20260929 10:55:26.520455 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 20487
I20260929 10:55:26.520531 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-0/wal/instance
I20260929 10:55:26.520689 20427 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.238.193:37641
I20260929 10:55:26.520737 11195 external_mini_cluster.cc:1428] Setting key 94b685e80ef584f38555731ef40d4d76
I20260929 10:55:26.521661 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.194:0
--local_ip_for_outbound_sockets=127.10.238.194
--webserver_interface=127.10.238.194
--webserver_port=0
--tserver_master_addrs=127.10.238.254:45401
--builtin_ntp_servers=127.10.238.212:38631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 10:55:26.603572 20618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:55:26.603776 20618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:55:26.603797 20618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:55:26.604941 20618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:55:26.604998 20618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:55:26.605023 20618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.194
I20260929 10:55:26.606503 20618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:38631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.238.194
--webserver_port=0
--tserver_master_addrs=127.10.238.254:45401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.194
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:55:26.606765 20618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:55:26.607057 20618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:55:26.612109 20626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:26.612154 20623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:26.612149 20624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:55:26.612444 20618 server_base.cc:1034] running on GCE node
I20260929 10:55:26.612661 20618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:55:26.612918 20618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:55:26.614058 20618 hybrid_clock.cc:648] HybridClock initialized: now 1790679326614048 us; error 32 us; skew 500 ppm
I20260929 10:55:26.615314 20618 webserver.cc:460] Webserver started at http://127.10.238.194:41115/ using document root <none> and password file <none>
I20260929 10:55:26.615551 20618 fs_manager.cc:359] Metadata directory not provided
I20260929 10:55:26.615597 20618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:55:26.615746 20618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:55:26.616849 20618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-1/data/instance:
uuid: "b25487404fa44c29bd711dd6d9f5d66d"
format_stamp: "Formatted at 2026-09-29 10:55:26 on dist-test-slave-8nrt"
server_key: "3aa5566cbdfd31cb69bc7b8e8483f508"
server_key_iv: "91e634b0d49e6a341edcaf2fca91e5b7"
server_key_version: "encryptionkey@0"
I20260929 10:55:26.617215 20618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-1/wal/instance:
uuid: "b25487404fa44c29bd711dd6d9f5d66d"
format_stamp: "Formatted at 2026-09-29 10:55:26 on dist-test-slave-8nrt"
server_key: "3aa5566cbdfd31cb69bc7b8e8483f508"
server_key_iv: "91e634b0d49e6a341edcaf2fca91e5b7"
server_key_version: "encryptionkey@0"
I20260929 10:55:26.618562 20618 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 10:55:26.619278 20632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:26.619469 20618 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 10:55:26.619567 20618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-1/wal
uuid: "b25487404fa44c29bd711dd6d9f5d66d"
format_stamp: "Formatted at 2026-09-29 10:55:26 on dist-test-slave-8nrt"
server_key: "3aa5566cbdfd31cb69bc7b8e8483f508"
server_key_iv: "91e634b0d49e6a341edcaf2fca91e5b7"
server_key_version: "encryptionkey@0"
I20260929 10:55:26.619655 20618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:55:26.638994 20618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:55:26.639320 20618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:55:26.639475 20618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:55:26.639703 20618 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:55:26.640046 20618 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:55:26.640102 20618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:26.640144 20618 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:55:26.640172 20618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:26.645949 20618 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.194:42835
I20260929 10:55:26.645992 20745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.194:42835 every 8 connection(s)
I20260929 10:55:26.646327 20618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
I20260929 10:55:26.646813 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 20618
I20260929 10:55:26.646911 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-1/wal/instance
I20260929 10:55:26.647132 11195 external_mini_cluster.cc:1428] Setting key 108f7c4697d71be1439651a4aea9df22
I20260929 10:55:26.648096 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.195:0
--local_ip_for_outbound_sockets=127.10.238.195
--webserver_interface=127.10.238.195
--webserver_port=0
--tserver_master_addrs=127.10.238.254:45401
--builtin_ntp_servers=127.10.238.212:38631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 10:55:26.651997 20467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:26.647088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:48613 (local address 127.10.238.254:45401)
0929 10:55:26.647133 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:26.647135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:26.647452 (+ 317us) server_negotiation.cc:407] Connection header received
0929 10:55:26.648087 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:26.648089 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:26.648101 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:26.648151 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:26.648934 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:26.649509 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:26.650513 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:26.650707 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:26.650783 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:26.651708 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:26.651710 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:26.651710 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:55:26.651715 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:26.651729 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:55:26.651733 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:26.651797 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:26.651892 (+ 95us) server_negotiation.cc:299] Negotiation successful
0929 10:55:26.651919 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 10:55:26.652278 20748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:26.647096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:45401 (local address 127.10.238.194:48613)
0929 10:55:26.647335 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:26.647381 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:26.647975 (+ 594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:26.648221 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:26.648226 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:26.648235 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:26.648837 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:26.648842 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:26.649606 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:26.649607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:26.650410 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:26.650412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:26.650793 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:26.650794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:26.650939 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:26.651555 (+ 616us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:55:26.651556 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:26.651584 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:55:26.651608 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:55:26.651812 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:55:26.651818 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:55:26.651821 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 10:55:26.651883 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 10:55:26.651987 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":85,"threads_started":1}
I20260929 10:55:26.652771 20746 heartbeater.cc:347] Connected to a master server at 127.10.238.254:45401
I20260929 10:55:26.652861 20746 heartbeater.cc:464] Registering TS with master...
I20260929 10:55:26.653061 20746 heartbeater.cc:511] Master 127.10.238.254:45401 requested a full tablet report, sending...
I20260929 10:55:26.653465 20427 ts_manager.cc:194] Registered new tserver with Master: b25487404fa44c29bd711dd6d9f5d66d (127.10.238.194:42835)
I20260929 10:55:26.654024 20427 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.238.194:48613
W20260929 10:55:26.730162 20749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:55:26.730396 20749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:55:26.730432 20749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:55:26.731565 20749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:55:26.731604 20749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:55:26.731629 20749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.195
I20260929 10:55:26.732885 20749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:38631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.238.195
--webserver_port=0
--tserver_master_addrs=127.10.238.254:45401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.195
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:55:26.733072 20749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:55:26.733399 20749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:55:26.738250 20754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:26.738296 20755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:55:26.738481 20749 server_base.cc:1034] running on GCE node
W20260929 10:55:26.738512 20757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:55:26.738790 20749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:55:26.739060 20749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:55:26.740224 20749 hybrid_clock.cc:648] HybridClock initialized: now 1790679326740215 us; error 41 us; skew 500 ppm
I20260929 10:55:26.741541 20749 webserver.cc:460] Webserver started at http://127.10.238.195:42153/ using document root <none> and password file <none>
I20260929 10:55:26.741796 20749 fs_manager.cc:359] Metadata directory not provided
I20260929 10:55:26.741875 20749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:55:26.742027 20749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:55:26.743149 20749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-2/data/instance:
uuid: "f49cdad566e94409b2b726e136d631af"
format_stamp: "Formatted at 2026-09-29 10:55:26 on dist-test-slave-8nrt"
server_key: "af3b58414634d58518b9244767748d23"
server_key_iv: "b2b6cf031e687ba3f3fb8501b427d46b"
server_key_version: "encryptionkey@0"
I20260929 10:55:26.743523 20749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-2/wal/instance:
uuid: "f49cdad566e94409b2b726e136d631af"
format_stamp: "Formatted at 2026-09-29 10:55:26 on dist-test-slave-8nrt"
server_key: "af3b58414634d58518b9244767748d23"
server_key_iv: "b2b6cf031e687ba3f3fb8501b427d46b"
server_key_version: "encryptionkey@0"
I20260929 10:55:26.744803 20749 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 10:55:26.745556 20763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:26.745719 20749 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 10:55:26.745829 20749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-2/wal
uuid: "f49cdad566e94409b2b726e136d631af"
format_stamp: "Formatted at 2026-09-29 10:55:26 on dist-test-slave-8nrt"
server_key: "af3b58414634d58518b9244767748d23"
server_key_iv: "b2b6cf031e687ba3f3fb8501b427d46b"
server_key_version: "encryptionkey@0"
I20260929 10:55:26.745904 20749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:55:26.788206 20749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:55:26.788550 20749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:55:26.788712 20749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:55:26.788971 20749 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:55:26.789448 20749 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:55:26.789505 20749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:26.789544 20749 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:55:26.789565 20749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:26.797039 20749 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.195:39455
I20260929 10:55:26.797120 20876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.195:39455 every 8 connection(s)
I20260929 10:55:26.797514 20749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
I20260929 10:55:26.803190 20467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:26.798380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:37825 (local address 127.10.238.254:45401)
0929 10:55:26.799185 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:26.799188 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:26.799201 (+ 13us) server_negotiation.cc:407] Connection header received
0929 10:55:26.799543 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:26.799544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:26.799559 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:26.799595 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:26.800440 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:26.800925 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:26.801971 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:26.802152 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:26.802207 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:26.802957 (+ 750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:26.802959 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:26.802959 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:55:26.802963 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:26.802978 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:55:26.802982 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:26.803027 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:26.803109 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 10:55:26.803130 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":774}
I20260929 10:55:26.803419 20879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:26.798394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:45401 (local address 127.10.238.195:37825)
0929 10:55:26.798673 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:26.798722 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:26.799406 (+ 684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:26.799820 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:26.799825 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:26.799833 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:26.800350 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:26.800355 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:26.801013 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:26.801014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:26.801897 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:26.801899 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:26.802226 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:26.802227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:26.802335 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:26.802818 (+ 483us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:55:26.802819 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:26.802847 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:55:26.802869 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:55:26.803039 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:55:26.803043 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:55:26.803046 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 10:55:26.803097 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 10:55:26.803175 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":108,"threads_started":1}
I20260929 10:55:26.803468 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 20749
I20260929 10:55:26.803552 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-2/wal/instance
I20260929 10:55:26.803736 11195 external_mini_cluster.cc:1428] Setting key 8511726b6c1effaf32930e6d4d5ea709
I20260929 10:55:26.804198 20877 heartbeater.cc:347] Connected to a master server at 127.10.238.254:45401
I20260929 10:55:26.804304 20877 heartbeater.cc:464] Registering TS with master...
I20260929 10:55:26.804505 20877 heartbeater.cc:511] Master 127.10.238.254:45401 requested a full tablet report, sending...
I20260929 10:55:26.804880 20426 ts_manager.cc:194] Registered new tserver with Master: f49cdad566e94409b2b726e136d631af (127.10.238.195:39455)
I20260929 10:55:26.805302 20426 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.238.195:37825
I20260929 10:55:26.815608 11195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 10:55:26.817618 20467 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:26.817009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38516 (local address 127.10.238.254:45401)
0929 10:55:26.817073 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:26.817074 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:26.817223 (+ 149us) server_negotiation.cc:407] Connection header received
0929 10:55:26.817318 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:26.817319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:26.817331 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:26.817393 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:26.817524 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38516: BlockingRecv error: recv got EOF from 127.0.0.1:38516 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 10:55:26.817716 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 20487
I20260929 10:55:26.822407 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 20618
I20260929 10:55:26.827060 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 20749
I20260929 10:55:26.831594 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 20396
2026-09-29T10:55:26Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (629 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T10:55:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:55:26Z Disabled control of system clock
I20260929 10:55:26.858825 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:36913
--webserver_interface=127.10.238.254
--webserver_port=0
--builtin_ntp_servers=127.10.238.212:35541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:36913
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 10:55:26.940151 20888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:55:26.940366 20888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:55:26.940388 20888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:55:26.941636 20888 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:55:26.941677 20888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:55:26.941690 20888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:55:26.941723 20888 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:55:26.941754 20888 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:55:26.943099 20888 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:35541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:36913
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:36913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:55:26.943364 20888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:55:26.943656 20888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:55:26.948462 20894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:26.948532 20893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:26.948525 20896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:55:26.948655 20888 server_base.cc:1034] running on GCE node
I20260929 10:55:26.948881 20888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:55:26.949182 20888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:55:26.950326 20888 hybrid_clock.cc:648] HybridClock initialized: now 1790679326950316 us; error 33 us; skew 500 ppm
I20260929 10:55:26.951522 20888 webserver.cc:460] Webserver started at http://127.10.238.254:33783/ using document root <none> and password file <none>
I20260929 10:55:26.951773 20888 fs_manager.cc:359] Metadata directory not provided
I20260929 10:55:26.951843 20888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:55:26.951992 20888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:55:26.953028 20888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/master-0/data/instance:
uuid: "b584d762fe75441e8694adf9d0cb3c66"
format_stamp: "Formatted at 2026-09-29 10:55:26 on dist-test-slave-8nrt"
server_key: "e05e537e29d2898e33302a2aeb4d7e2c"
server_key_iv: "754a46e49a5dab052ced72ce3d4289a1"
server_key_version: "encryptionkey@0"
I20260929 10:55:26.953433 20888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/master-0/wal/instance:
uuid: "b584d762fe75441e8694adf9d0cb3c66"
format_stamp: "Formatted at 2026-09-29 10:55:26 on dist-test-slave-8nrt"
server_key: "e05e537e29d2898e33302a2aeb4d7e2c"
server_key_iv: "754a46e49a5dab052ced72ce3d4289a1"
server_key_version: "encryptionkey@0"
I20260929 10:55:26.954794 20888 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 10:55:26.955504 20902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:26.955679 20888 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 10:55:26.955767 20888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "b584d762fe75441e8694adf9d0cb3c66"
format_stamp: "Formatted at 2026-09-29 10:55:26 on dist-test-slave-8nrt"
server_key: "e05e537e29d2898e33302a2aeb4d7e2c"
server_key_iv: "754a46e49a5dab052ced72ce3d4289a1"
server_key_version: "encryptionkey@0"
I20260929 10:55:26.955850 20888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:55:26.959141 20888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:55:26.959408 20888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:55:26.959544 20888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:55:26.963466 20888 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.254:36913
I20260929 10:55:26.963495 20954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.254:36913 every 8 connection(s)
I20260929 10:55:26.963817 20888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:55:26.964439 20955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:55:26.967142 20955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b584d762fe75441e8694adf9d0cb3c66: Bootstrap starting.
I20260929 10:55:26.967717 20955 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b584d762fe75441e8694adf9d0cb3c66: Neither blocks nor log segments found. Creating new log.
I20260929 10:55:26.968026 20955 log.cc:824] T 00000000000000000000000000000000 P b584d762fe75441e8694adf9d0cb3c66: Log is configured to *not* fsync() on all Append() calls
I20260929 10:55:26.968698 20955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b584d762fe75441e8694adf9d0cb3c66: No bootstrap required, opened a new log
I20260929 10:55:26.970376 20955 raft_consensus.cc:348] T 00000000000000000000000000000000 P b584d762fe75441e8694adf9d0cb3c66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b584d762fe75441e8694adf9d0cb3c66" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 36913 } }
I20260929 10:55:26.970515 20955 raft_consensus.cc:374] T 00000000000000000000000000000000 P b584d762fe75441e8694adf9d0cb3c66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:55:26.970562 20955 raft_consensus.cc:729] T 00000000000000000000000000000000 P b584d762fe75441e8694adf9d0cb3c66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b584d762fe75441e8694adf9d0cb3c66, State: Initialized, Role: FOLLOWER
I20260929 10:55:26.970717 20955 consensus_queue.cc:260] T 00000000000000000000000000000000 P b584d762fe75441e8694adf9d0cb3c66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b584d762fe75441e8694adf9d0cb3c66" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 36913 } }
I20260929 10:55:26.970779 20955 raft_consensus.cc:388] T 00000000000000000000000000000000 P b584d762fe75441e8694adf9d0cb3c66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:55:26.970800 20955 raft_consensus.cc:482] T 00000000000000000000000000000000 P b584d762fe75441e8694adf9d0cb3c66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:55:26.970866 20955 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b584d762fe75441e8694adf9d0cb3c66 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:55:26.971414 20955 raft_consensus.cc:504] T 00000000000000000000000000000000 P b584d762fe75441e8694adf9d0cb3c66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b584d762fe75441e8694adf9d0cb3c66" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 36913 } }
I20260929 10:55:26.971524 20955 leader_election.cc:302] T 00000000000000000000000000000000 P b584d762fe75441e8694adf9d0cb3c66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b584d762fe75441e8694adf9d0cb3c66; no voters:
I20260929 10:55:26.971704 20955 leader_election.cc:288] T 00000000000000000000000000000000 P b584d762fe75441e8694adf9d0cb3c66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:55:26.971850 20958 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b584d762fe75441e8694adf9d0cb3c66 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:55:26.971957 20955 sys_catalog.cc:564] T 00000000000000000000000000000000 P b584d762fe75441e8694adf9d0cb3c66 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:55:26.972070 20958 raft_consensus.cc:686] T 00000000000000000000000000000000 P b584d762fe75441e8694adf9d0cb3c66 [term 1 LEADER]: Becoming Leader. State: Replica: b584d762fe75441e8694adf9d0cb3c66, State: Running, Role: LEADER
I20260929 10:55:26.972184 20958 consensus_queue.cc:237] T 00000000000000000000000000000000 P b584d762fe75441e8694adf9d0cb3c66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b584d762fe75441e8694adf9d0cb3c66" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 36913 } }
I20260929 10:55:26.972958 20960 sys_catalog.cc:455] T 00000000000000000000000000000000 P b584d762fe75441e8694adf9d0cb3c66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b584d762fe75441e8694adf9d0cb3c66. Latest consensus state: current_term: 1 leader_uuid: "b584d762fe75441e8694adf9d0cb3c66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b584d762fe75441e8694adf9d0cb3c66" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 36913 } } }
I20260929 10:55:26.973027 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 20888
I20260929 10:55:26.973061 20960 sys_catalog.cc:458] T 00000000000000000000000000000000 P b584d762fe75441e8694adf9d0cb3c66 [sys.catalog]: This master's current role is: LEADER
I20260929 10:55:26.973147 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/master-0/wal/instance
I20260929 10:55:26.973351 20970 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:55:26.973441 11195 external_mini_cluster.cc:1428] Setting key ca74795403f8a3a4191a0000c1675406
I20260929 10:55:26.973737 20959 sys_catalog.cc:455] T 00000000000000000000000000000000 P b584d762fe75441e8694adf9d0cb3c66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b584d762fe75441e8694adf9d0cb3c66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b584d762fe75441e8694adf9d0cb3c66" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 36913 } } }
I20260929 10:55:26.974284 20959 sys_catalog.cc:458] T 00000000000000000000000000000000 P b584d762fe75441e8694adf9d0cb3c66 [sys.catalog]: This master's current role is: LEADER
I20260929 10:55:26.974363 20970 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:55:26.975873 20970 catalog_manager.cc:1347] Generated new cluster ID: 9903f607422b4b3ab556f81bb29ac739
I20260929 10:55:26.975916 20970 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:55:26.979035 20977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:26.974028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57920 (local address 127.10.238.254:36913)
0929 10:55:26.977444 (+ 3416us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:26.977454 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:26.977494 (+ 40us) server_negotiation.cc:407] Connection header received
0929 10:55:26.977633 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:26.977641 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:26.977688 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:26.977860 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:26.978542 (+ 682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:26.978546 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:26.978547 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:55:26.978552 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:26.978600 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:55:26.978615 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:26.978701 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:26.978769 (+ 68us) server_negotiation.cc:299] Negotiation successful
0929 10:55:26.978834 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3318,"thread_start_us":73,"threads_started":1}
I20260929 10:55:26.981947 20970 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:55:26.982098 20970 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:55:26.985968 20970 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b584d762fe75441e8694adf9d0cb3c66: Generated new TSK 0
I20260929 10:55:26.986166 20970 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:55:26.991745 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.193:0
--local_ip_for_outbound_sockets=127.10.238.193
--webserver_interface=127.10.238.193
--webserver_port=0
--tserver_master_addrs=127.10.238.254:36913
--builtin_ntp_servers=127.10.238.212:35541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 10:55:27.075820 20979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:55:27.076050 20979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:55:27.076086 20979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:55:27.077363 20979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:55:27.077425 20979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:55:27.077466 20979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.193
I20260929 10:55:27.078936 20979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:35541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.238.193
--webserver_port=0
--tserver_master_addrs=127.10.238.254:36913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.193
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:55:27.079205 20979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:55:27.079511 20979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:55:27.084609 20984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:27.084649 20987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:27.084659 20985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:55:27.084921 20979 server_base.cc:1034] running on GCE node
I20260929 10:55:27.085140 20979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:55:27.085438 20979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:55:27.086611 20979 hybrid_clock.cc:648] HybridClock initialized: now 1790679327086591 us; error 36 us; skew 500 ppm
I20260929 10:55:27.088043 20979 webserver.cc:460] Webserver started at http://127.10.238.193:45951/ using document root <none> and password file <none>
I20260929 10:55:27.088306 20979 fs_manager.cc:359] Metadata directory not provided
I20260929 10:55:27.088352 20979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:55:27.088478 20979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:55:27.089624 20979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-0/data/instance:
uuid: "611a2f49fe1c484384ddbd53f6b419a2"
format_stamp: "Formatted at 2026-09-29 10:55:27 on dist-test-slave-8nrt"
server_key: "7195421ee134d3ba392074d69974ae73"
server_key_iv: "9e1b3d8043a26d5cc1442dcdd5feadd3"
server_key_version: "encryptionkey@0"
I20260929 10:55:27.089982 20979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-0/wal/instance:
uuid: "611a2f49fe1c484384ddbd53f6b419a2"
format_stamp: "Formatted at 2026-09-29 10:55:27 on dist-test-slave-8nrt"
server_key: "7195421ee134d3ba392074d69974ae73"
server_key_iv: "9e1b3d8043a26d5cc1442dcdd5feadd3"
server_key_version: "encryptionkey@0"
I20260929 10:55:27.091331 20979 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 10:55:27.092150 20993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:27.092356 20979 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:55:27.092454 20979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-0/wal
uuid: "611a2f49fe1c484384ddbd53f6b419a2"
format_stamp: "Formatted at 2026-09-29 10:55:27 on dist-test-slave-8nrt"
server_key: "7195421ee134d3ba392074d69974ae73"
server_key_iv: "9e1b3d8043a26d5cc1442dcdd5feadd3"
server_key_version: "encryptionkey@0"
I20260929 10:55:27.092541 20979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:55:27.095943 20979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:55:27.096230 20979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:55:27.096374 20979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:55:27.096628 20979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:55:27.096995 20979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:55:27.097048 20979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:27.097088 20979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:55:27.097115 20979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:27.104341 20979 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.193:44519
I20260929 10:55:27.104374 21106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.193:44519 every 8 connection(s)
I20260929 10:55:27.104714 20979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
I20260929 10:55:27.106720 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 20979
I20260929 10:55:27.106819 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-0/wal/instance
I20260929 10:55:27.107030 11195 external_mini_cluster.cc:1428] Setting key 5bbf6834cb1ef990130a5efcb35e8459
I20260929 10:55:27.107954 20977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:27.105541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:60293 (local address 127.10.238.254:36913)
0929 10:55:27.105605 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:27.105608 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:27.105844 (+ 236us) server_negotiation.cc:407] Connection header received
0929 10:55:27.106307 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:27.106309 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:27.106318 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:27.106433 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:27.107727 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:27.107728 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:27.107728 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:55:27.107733 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:27.107750 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:55:27.107755 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:27.107801 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:27.107873 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 10:55:27.107887 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 10:55:27.108165 21109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:27.105524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:36913 (local address 127.10.238.193:60293)
0929 10:55:27.105710 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:27.105750 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:27.106208 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:27.106478 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:27.106481 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:27.106489 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:27.107574 (+ 1085us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:55:27.107575 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:27.107603 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:55:27.107630 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:55:27.107809 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:55:27.107814 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:55:27.107817 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 10:55:27.107885 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 10:55:27.107946 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
I20260929 10:55:27.108188 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.194:0
--local_ip_for_outbound_sockets=127.10.238.194
--webserver_interface=127.10.238.194
--webserver_port=0
--tserver_master_addrs=127.10.238.254:36913
--builtin_ntp_servers=127.10.238.212:35541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 10:55:27.110059 21107 heartbeater.cc:347] Connected to a master server at 127.10.238.254:36913
I20260929 10:55:27.110167 21107 heartbeater.cc:464] Registering TS with master...
I20260929 10:55:27.110320 21107 heartbeater.cc:511] Master 127.10.238.254:36913 requested a full tablet report, sending...
I20260929 10:55:27.110888 20919 ts_manager.cc:194] Registered new tserver with Master: 611a2f49fe1c484384ddbd53f6b419a2 (127.10.238.193:44519)
W20260929 10:55:27.194794 21110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:55:27.195031 21110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:55:27.195056 21110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:55:27.196206 21110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:55:27.196249 21110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:55:27.196275 21110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.194
I20260929 10:55:27.197609 21110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:35541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.238.194
--webserver_port=0
--tserver_master_addrs=127.10.238.254:36913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.194
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:55:27.197813 21110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:55:27.198071 21110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:55:27.203153 21115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:27.203137 21118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:27.203137 21116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:55:27.203390 21110 server_base.cc:1034] running on GCE node
I20260929 10:55:27.203624 21110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:55:27.203929 21110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:55:27.205083 21110 hybrid_clock.cc:648] HybridClock initialized: now 1790679327205070 us; error 33 us; skew 500 ppm
I20260929 10:55:27.206463 21110 webserver.cc:460] Webserver started at http://127.10.238.194:46171/ using document root <none> and password file <none>
I20260929 10:55:27.206737 21110 fs_manager.cc:359] Metadata directory not provided
I20260929 10:55:27.206794 21110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:55:27.206948 21110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:55:27.208078 21110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-1/data/instance:
uuid: "a63da414332a4707a07d28eecb61378b"
format_stamp: "Formatted at 2026-09-29 10:55:27 on dist-test-slave-8nrt"
server_key: "3e171e269fd0546961780ddc740b1b14"
server_key_iv: "19524da6df255187efe25475761887aa"
server_key_version: "encryptionkey@0"
I20260929 10:55:27.208453 21110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-1/wal/instance:
uuid: "a63da414332a4707a07d28eecb61378b"
format_stamp: "Formatted at 2026-09-29 10:55:27 on dist-test-slave-8nrt"
server_key: "3e171e269fd0546961780ddc740b1b14"
server_key_iv: "19524da6df255187efe25475761887aa"
server_key_version: "encryptionkey@0"
I20260929 10:55:27.209848 21110 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260929 10:55:27.210554 21124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:27.210731 21110 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:27.210826 21110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-1/wal
uuid: "a63da414332a4707a07d28eecb61378b"
format_stamp: "Formatted at 2026-09-29 10:55:27 on dist-test-slave-8nrt"
server_key: "3e171e269fd0546961780ddc740b1b14"
server_key_iv: "19524da6df255187efe25475761887aa"
server_key_version: "encryptionkey@0"
I20260929 10:55:27.210903 21110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:55:27.214318 21110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:55:27.214572 21110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:55:27.214704 21110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:55:27.214906 21110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:55:27.215195 21110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:55:27.215226 21110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:27.215286 21110 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:55:27.215313 21110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:27.221083 21110 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.194:33131
I20260929 10:55:27.221143 21237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.194:33131 every 8 connection(s)
I20260929 10:55:27.221451 21110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
I20260929 10:55:27.224327 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 21110
I20260929 10:55:27.224465 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-1/wal/instance
I20260929 10:55:27.224526 20977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:27.222224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:54239 (local address 127.10.238.254:36913)
0929 10:55:27.222278 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:27.222280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:27.222531 (+ 251us) server_negotiation.cc:407] Connection header received
0929 10:55:27.222971 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:27.222972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:27.222977 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:27.223031 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:27.224221 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:27.224223 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:27.224223 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:55:27.224228 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:27.224252 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:55:27.224258 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:27.224326 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:27.224418 (+ 92us) server_negotiation.cc:299] Negotiation successful
0929 10:55:27.224437 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 10:55:27.224670 11195 external_mini_cluster.cc:1428] Setting key 143d340cb5fa7e434b5227f65e21313e
I20260929 10:55:27.224738 21240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:27.222223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:36913 (local address 127.10.238.194:54239)
0929 10:55:27.222413 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:27.222446 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:27.222891 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:27.223081 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:27.223084 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:27.223092 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:27.224056 (+ 964us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:55:27.224056 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:27.224085 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:55:27.224133 (+ 48us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:55:27.224339 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:55:27.224343 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:55:27.224345 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 10:55:27.224399 (+ 54us) client_negotiation.cc:240] Negotiation successful
0929 10:55:27.224487 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":89,"threads_started":1}
I20260929 10:55:27.225651 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.195:0
--local_ip_for_outbound_sockets=127.10.238.195
--webserver_interface=127.10.238.195
--webserver_port=0
--tserver_master_addrs=127.10.238.254:36913
--builtin_ntp_servers=127.10.238.212:35541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 10:55:27.226007 21238 heartbeater.cc:347] Connected to a master server at 127.10.238.254:36913
I20260929 10:55:27.226110 21238 heartbeater.cc:464] Registering TS with master...
I20260929 10:55:27.226274 21238 heartbeater.cc:511] Master 127.10.238.254:36913 requested a full tablet report, sending...
I20260929 10:55:27.226629 20919 ts_manager.cc:194] Registered new tserver with Master: a63da414332a4707a07d28eecb61378b (127.10.238.194:33131)
W20260929 10:55:27.308408 21241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:55:27.308634 21241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:55:27.308657 21241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:55:27.309899 21241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:55:27.309948 21241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:55:27.309975 21241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.195
I20260929 10:55:27.311365 21241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:35541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.238.195
--webserver_port=0
--tserver_master_addrs=127.10.238.254:36913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.195
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:55:27.311636 21241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:55:27.311970 21241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:55:27.317198 21247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:55:27.317373 21241 server_base.cc:1034] running on GCE node
W20260929 10:55:27.317210 21249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:27.317217 21246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:55:27.317658 21241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:55:27.317929 21241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:55:27.319087 21241 hybrid_clock.cc:648] HybridClock initialized: now 1790679327319080 us; error 54 us; skew 500 ppm
I20260929 10:55:27.320360 21241 webserver.cc:460] Webserver started at http://127.10.238.195:39695/ using document root <none> and password file <none>
I20260929 10:55:27.320627 21241 fs_manager.cc:359] Metadata directory not provided
I20260929 10:55:27.320683 21241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:55:27.320889 21241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:55:27.322034 21241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-2/data/instance:
uuid: "ff707f7c410e410fb07d3df76055a4c2"
format_stamp: "Formatted at 2026-09-29 10:55:27 on dist-test-slave-8nrt"
server_key: "f872874fc4ee0ea2616fc0b5d245a20e"
server_key_iv: "3df9e5771b87bc42c9598eb914ae0b77"
server_key_version: "encryptionkey@0"
I20260929 10:55:27.322399 21241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-2/wal/instance:
uuid: "ff707f7c410e410fb07d3df76055a4c2"
format_stamp: "Formatted at 2026-09-29 10:55:27 on dist-test-slave-8nrt"
server_key: "f872874fc4ee0ea2616fc0b5d245a20e"
server_key_iv: "3df9e5771b87bc42c9598eb914ae0b77"
server_key_version: "encryptionkey@0"
I20260929 10:55:27.323694 21241 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 10:55:27.324429 21255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:27.324606 21241 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:55:27.324712 21241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-2/wal
uuid: "ff707f7c410e410fb07d3df76055a4c2"
format_stamp: "Formatted at 2026-09-29 10:55:27 on dist-test-slave-8nrt"
server_key: "f872874fc4ee0ea2616fc0b5d245a20e"
server_key_iv: "3df9e5771b87bc42c9598eb914ae0b77"
server_key_version: "encryptionkey@0"
I20260929 10:55:27.324815 21241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:55:27.328225 21241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:55:27.328500 21241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:55:27.328648 21241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:55:27.328887 21241 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:55:27.329205 21241 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:55:27.329285 21241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:27.329329 21241 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:55:27.329355 21241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:27.335245 21241 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.195:38243
I20260929 10:55:27.335297 21368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.195:38243 every 8 connection(s)
I20260929 10:55:27.335623 21241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
I20260929 10:55:27.338639 20977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:27.336327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:33339 (local address 127.10.238.254:36913)
0929 10:55:27.336417 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:27.336420 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:27.336636 (+ 216us) server_negotiation.cc:407] Connection header received
0929 10:55:27.337134 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:27.337135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:27.337140 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:27.337196 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:27.338450 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:27.338451 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:27.338451 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:55:27.338454 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:27.338466 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:55:27.338470 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:27.338515 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:27.338574 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 10:55:27.338586 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 10:55:27.338812 21371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:27.336288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:36913 (local address 127.10.238.195:33339)
0929 10:55:27.336505 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:27.336539 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:27.337035 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:27.337250 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:27.337300 (+ 50us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:27.337310 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:27.338311 (+ 1001us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:55:27.338312 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:27.338341 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:55:27.338370 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:55:27.338512 (+ 142us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:55:27.338515 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:55:27.338517 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 10:55:27.338568 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 10:55:27.338615 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":118,"threads_started":1}
I20260929 10:55:27.339385 21369 heartbeater.cc:347] Connected to a master server at 127.10.238.254:36913
I20260929 10:55:27.339485 21369 heartbeater.cc:464] Registering TS with master...
I20260929 10:55:27.339632 21369 heartbeater.cc:511] Master 127.10.238.254:36913 requested a full tablet report, sending...
I20260929 10:55:27.339993 20919 ts_manager.cc:194] Registered new tserver with Master: ff707f7c410e410fb07d3df76055a4c2 (127.10.238.195:38243)
I20260929 10:55:27.340574 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 21241
I20260929 10:55:27.340647 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790679274510869-11195-0/minicluster-data/ts-2/wal/instance
I20260929 10:55:27.340797 11195 external_mini_cluster.cc:1428] Setting key d258ad65eec424884b45ea9ff86f8824
I20260929 10:55:27.341637 11195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 10:55:27.343313 20977 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:55:27.342818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57932 (local address 127.10.238.254:36913)
0929 10:55:27.342884 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:27.342885 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:27.343030 (+ 145us) server_negotiation.cc:407] Connection header received
0929 10:55:27.343056 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:27.343056 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:27.343061 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:27.343111 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:27.343238 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57932: BlockingRecv error: recv got EOF from 127.0.0.1:57932 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 10:55:27.343387 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 20979
I20260929 10:55:27.348730 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 21110
I20260929 10:55:27.353681 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 21241
I20260929 10:55:27.358080 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 20888
2026-09-29T10:55:27Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (525 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:55:27 dist-test-slave-8nrt krb5kdc[21379](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:55:27 dist-test-slave-8nrt krb5kdc[21379](info): set up 2 sockets
Sep 29 10:55:27 dist-test-slave-8nrt krb5kdc[21379](info): commencing operation
krb5kdc: starting...
W20260929 10:55:28.402225 11195 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Sep 29 10:55:28 dist-test-slave-8nrt krb5kdc[21379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:55:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:55:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
I20260929 10:55:28.520432 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:45429
--webserver_interface=127.10.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:35813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:45429
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:55:28.601543 21392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:55:28.601763 21392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:55:28.601785 21392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:55:28.602892 21392 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:55:28.602928 21392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:55:28.602942 21392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:55:28.602955 21392 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:55:28.602967 21392 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:55:28.604367 21392 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:35813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:45429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:45429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:55:28.604559 21392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:55:28.604737 21392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:55:28.609516 21397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:28.609541 21400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:28.609566 21398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:55:28.609850 21392 server_base.cc:1034] running on GCE node
I20260929 10:55:28.610114 21392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:55:28.610404 21392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:55:28.611608 21392 hybrid_clock.cc:648] HybridClock initialized: now 1790679328611604 us; error 42 us; skew 500 ppm
Sep 29 10:55:28 dist-test-slave-8nrt krb5kdc[21379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679328, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:55:28.614246 21392 init.cc:377] Logged in from keytab as kudu/127.10.238.254@KRBTEST.COM (short username kudu)
I20260929 10:55:28.614686 21392 webserver.cc:460] Webserver started at http://127.10.238.254:43941/ using document root <none> and password file <none>
I20260929 10:55:28.614924 21392 fs_manager.cc:359] Metadata directory not provided
I20260929 10:55:28.614995 21392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:55:28.615155 21392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:55:28.616263 21392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/master-0/data/instance:
uuid: "59815ec992df4f5eb6b399f54a5888fe"
format_stamp: "Formatted at 2026-09-29 10:55:28 on dist-test-slave-8nrt"
server_key: "cb3b4050abce636129af194c4d16841e"
server_key_iv: "88aee5ae30bf96344ccdc1a116982c70"
server_key_version: "encryptionkey@0"
I20260929 10:55:28.616649 21392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/master-0/wal/instance:
uuid: "59815ec992df4f5eb6b399f54a5888fe"
format_stamp: "Formatted at 2026-09-29 10:55:28 on dist-test-slave-8nrt"
server_key: "cb3b4050abce636129af194c4d16841e"
server_key_iv: "88aee5ae30bf96344ccdc1a116982c70"
server_key_version: "encryptionkey@0"
I20260929 10:55:28.618083 21392 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 10:55:28.618870 21407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:28.619037 21392 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 10:55:28.619143 21392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "59815ec992df4f5eb6b399f54a5888fe"
format_stamp: "Formatted at 2026-09-29 10:55:28 on dist-test-slave-8nrt"
server_key: "cb3b4050abce636129af194c4d16841e"
server_key_iv: "88aee5ae30bf96344ccdc1a116982c70"
server_key_version: "encryptionkey@0"
I20260929 10:55:28.619218 21392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:55:28.633091 21392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:55:28.634984 21392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:55:28.635154 21392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:55:28.639127 21392 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.254:45429
I20260929 10:55:28.639173 21459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.254:45429 every 8 connection(s)
I20260929 10:55:28.639544 21392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:55:28.640554 21460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:55:28.642769 21460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59815ec992df4f5eb6b399f54a5888fe: Bootstrap starting.
I20260929 10:55:28.643283 21460 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59815ec992df4f5eb6b399f54a5888fe: Neither blocks nor log segments found. Creating new log.
I20260929 10:55:28.643532 21460 log.cc:824] T 00000000000000000000000000000000 P 59815ec992df4f5eb6b399f54a5888fe: Log is configured to *not* fsync() on all Append() calls
I20260929 10:55:28.644189 21460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59815ec992df4f5eb6b399f54a5888fe: No bootstrap required, opened a new log
I20260929 10:55:28.645459 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 21392
I20260929 10:55:28.645565 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/master-0/wal/instance
I20260929 10:55:28.645761 11195 external_mini_cluster.cc:1428] Setting key e1116a7a81e4494b03853366673cae34
I20260929 10:55:28.646160 21460 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59815ec992df4f5eb6b399f54a5888fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59815ec992df4f5eb6b399f54a5888fe" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 45429 } }
I20260929 10:55:28.646364 21460 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59815ec992df4f5eb6b399f54a5888fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:55:28.646404 21460 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59815ec992df4f5eb6b399f54a5888fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59815ec992df4f5eb6b399f54a5888fe, State: Initialized, Role: FOLLOWER
I20260929 10:55:28.646557 21460 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59815ec992df4f5eb6b399f54a5888fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59815ec992df4f5eb6b399f54a5888fe" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 45429 } }
I20260929 10:55:28.646643 21460 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59815ec992df4f5eb6b399f54a5888fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:55:28.646677 21460 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59815ec992df4f5eb6b399f54a5888fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:55:28.646726 21460 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59815ec992df4f5eb6b399f54a5888fe [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:55:28.647531 21460 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59815ec992df4f5eb6b399f54a5888fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59815ec992df4f5eb6b399f54a5888fe" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 45429 } }
I20260929 10:55:28.647650 21460 leader_election.cc:302] T 00000000000000000000000000000000 P 59815ec992df4f5eb6b399f54a5888fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59815ec992df4f5eb6b399f54a5888fe; no voters:
I20260929 10:55:28.647817 21460 leader_election.cc:288] T 00000000000000000000000000000000 P 59815ec992df4f5eb6b399f54a5888fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:55:28.647887 21465 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59815ec992df4f5eb6b399f54a5888fe [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:55:28.648041 21465 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59815ec992df4f5eb6b399f54a5888fe [term 1 LEADER]: Becoming Leader. State: Replica: 59815ec992df4f5eb6b399f54a5888fe, State: Running, Role: LEADER
I20260929 10:55:28.648068 21460 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59815ec992df4f5eb6b399f54a5888fe [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:55:28.648137 21465 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59815ec992df4f5eb6b399f54a5888fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59815ec992df4f5eb6b399f54a5888fe" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 45429 } }
I20260929 10:55:28.648469 21465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59815ec992df4f5eb6b399f54a5888fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59815ec992df4f5eb6b399f54a5888fe. Latest consensus state: current_term: 1 leader_uuid: "59815ec992df4f5eb6b399f54a5888fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59815ec992df4f5eb6b399f54a5888fe" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 45429 } } }
I20260929 10:55:28.648557 21465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59815ec992df4f5eb6b399f54a5888fe [sys.catalog]: This master's current role is: LEADER
I20260929 10:55:28.648480 21466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59815ec992df4f5eb6b399f54a5888fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59815ec992df4f5eb6b399f54a5888fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59815ec992df4f5eb6b399f54a5888fe" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 45429 } } }
I20260929 10:55:28.648753 21466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59815ec992df4f5eb6b399f54a5888fe [sys.catalog]: This master's current role is: LEADER
I20260929 10:55:28.648859 21472 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:55:28.649420 21472 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:55:28.652280 21472 catalog_manager.cc:1347] Generated new cluster ID: cae9ca96097f472488701e7e0d0d9240
I20260929 10:55:28.652338 21472 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 10:55:28 dist-test-slave-8nrt krb5kdc[21379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:55:28.657611 21464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:28.646378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47992 (local address 127.10.238.254:45429)
0929 10:55:28.646559 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:28.646569 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:28.646608 (+ 39us) server_negotiation.cc:407] Connection header received
0929 10:55:28.646776 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:28.646784 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:28.646841 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:28.646959 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:28.647742 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:28.649295 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:28.650491 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:28.650895 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:28.651015 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:28.653777 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:28.653781 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:28.653781 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:28.653794 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:28.655859 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:28.656243 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:28.656253 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:28.656254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:28.656405 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:28.656623 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:28.656624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:28.656624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:28.656971 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:28.657148 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:28.657219 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 10:55:28.657403 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":95,"threads_started":1}
I20260929 10:55:28.664790 21472 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:55:28.665575 21472 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:55:28.674047 21472 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59815ec992df4f5eb6b399f54a5888fe: Generated new TSK 0
I20260929 10:55:28.674268 21472 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.238.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab.
Entry for principal kudu/127.10.238.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab.
I20260929 10:55:28.706283 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.193:0
--local_ip_for_outbound_sockets=127.10.238.193
--webserver_interface=127.10.238.193
--webserver_port=0
--tserver_master_addrs=127.10.238.254:45429
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab
--principal=kudu/127.10.238.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:35813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:55:28.790272 21486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:55:28.790592 21486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:55:28.790634 21486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:55:28.792845 21486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:55:28.792949 21486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:55:28.793004 21486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.193
I20260929 10:55:28.795118 21486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:35813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab
--principal=kudu/127.10.238.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.238.193
--webserver_port=0
--tserver_master_addrs=127.10.238.254:45429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.193
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:55:28.795437 21486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:55:28.795773 21486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:55:28.801290 21491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:28.801344 21492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:28.801403 21494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:55:28.801656 21486 server_base.cc:1034] running on GCE node
I20260929 10:55:28.801911 21486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:55:28.802207 21486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:55:28.803388 21486 hybrid_clock.cc:648] HybridClock initialized: now 1790679328803381 us; error 54 us; skew 500 ppm
Sep 29 10:55:28 dist-test-slave-8nrt krb5kdc[21379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679328, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:55:28.805997 21486 init.cc:377] Logged in from keytab as kudu/127.10.238.193@KRBTEST.COM (short username kudu)
I20260929 10:55:28.806476 21486 webserver.cc:460] Webserver started at http://127.10.238.193:34537/ using document root <none> and password file <none>
I20260929 10:55:28.806671 21486 fs_manager.cc:359] Metadata directory not provided
I20260929 10:55:28.806712 21486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:55:28.806810 21486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:55:28.807829 21486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/ts-0/data/instance:
uuid: "d616379489274a46928923186f4248ed"
format_stamp: "Formatted at 2026-09-29 10:55:28 on dist-test-slave-8nrt"
server_key: "dc73670bd00302d3392390444719c9f2"
server_key_iv: "3e2de4f5c3527ba786f2ec7ba4e5b7b2"
server_key_version: "encryptionkey@0"
I20260929 10:55:28.808125 21486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/ts-0/wal/instance:
uuid: "d616379489274a46928923186f4248ed"
format_stamp: "Formatted at 2026-09-29 10:55:28 on dist-test-slave-8nrt"
server_key: "dc73670bd00302d3392390444719c9f2"
server_key_iv: "3e2de4f5c3527ba786f2ec7ba4e5b7b2"
server_key_version: "encryptionkey@0"
I20260929 10:55:28.809464 21486 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 10:55:28.810139 21501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:28.810348 21486 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:55:28.810432 21486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/ts-0/wal
uuid: "d616379489274a46928923186f4248ed"
format_stamp: "Formatted at 2026-09-29 10:55:28 on dist-test-slave-8nrt"
server_key: "dc73670bd00302d3392390444719c9f2"
server_key_iv: "3e2de4f5c3527ba786f2ec7ba4e5b7b2"
server_key_version: "encryptionkey@0"
I20260929 10:55:28.810489 21486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:55:28.855679 21486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:55:28.858194 21486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:55:28.858325 21486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:55:28.858522 21486 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:55:28.858875 21486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:55:28.858912 21486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:28.858935 21486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:55:28.858947 21486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:28.864836 21486 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.193:41053
I20260929 10:55:28.864923 21614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.193:41053 every 8 connection(s)
I20260929 10:55:28.865270 21486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
Sep 29 10:55:28 dist-test-slave-8nrt krb5kdc[21379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679328, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.193@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:55:28.872071 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 21486
I20260929 10:55:28.872176 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/ts-0/wal/instance
I20260929 10:55:28.872453 11195 external_mini_cluster.cc:1428] Setting key f6594d21fa2928f91309ba6e6d33e3d8
I20260929 10:55:28.875151 21464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:28.866074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:35889 (local address 127.10.238.254:45429)
0929 10:55:28.866161 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:28.866164 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:28.866355 (+ 191us) server_negotiation.cc:407] Connection header received
0929 10:55:28.866860 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:28.866861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:28.866876 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:28.866930 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:28.867862 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:28.868368 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:28.869440 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:28.869633 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:28.869696 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:28.871897 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:28.871899 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:28.871899 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:28.871913 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:28.873738 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:28.874127 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:28.874129 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:28.874129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:28.874193 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:28.874431 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:28.874432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:28.874432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:28.874582 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:28.874658 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:28.875013 (+ 355us) server_negotiation.cc:299] Negotiation successful
0929 10:55:28.875087 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 10:55:28.875461 21617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:28.866031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:45429 (local address 127.10.238.193:35889)
0929 10:55:28.866226 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:28.866264 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:28.866758 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:28.866987 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:28.866990 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:28.867563 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:28.867765 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:28.867770 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:28.868461 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:28.868462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:28.869352 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:28.869355 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:28.869707 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:28.869708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:28.869825 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:28.870376 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:28.870385 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:28.871752 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:55:28.873825 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:28.873832 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:28.873834 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:28.874055 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:28.874260 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:28.874261 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:28.874261 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:28.874367 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:28.874666 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:55:28.874673 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:55:28.874852 (+ 179us) client_negotiation.cc:769] Sending connection context
0929 10:55:28.874984 (+ 132us) client_negotiation.cc:240] Negotiation successful
0929 10:55:28.875173 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":115,"threads_started":1}
I20260929 10:55:28.876060 21615 heartbeater.cc:347] Connected to a master server at 127.10.238.254:45429
I20260929 10:55:28.876168 21615 heartbeater.cc:464] Registering TS with master...
I20260929 10:55:28.876405 21615 heartbeater.cc:511] Master 127.10.238.254:45429 requested a full tablet report, sending...
I20260929 10:55:28.876982 21424 ts_manager.cc:194] Registered new tserver with Master: d616379489274a46928923186f4248ed (127.10.238.193:41053)
I20260929 10:55:28.877693 21424 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.193@KRBTEST.COM'} at 127.10.238.193:35889
WARNING: no policy specified for kudu/127.10.238.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab.
Entry for principal kudu/127.10.238.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab.
I20260929 10:55:28.900121 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.194:0
--local_ip_for_outbound_sockets=127.10.238.194
--webserver_interface=127.10.238.194
--webserver_port=0
--tserver_master_addrs=127.10.238.254:45429
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab
--principal=kudu/127.10.238.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:35813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:55:28.982436 21620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:55:28.982643 21620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:55:28.982667 21620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:55:28.983754 21620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:55:28.983799 21620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:55:28.983825 21620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.194
I20260929 10:55:28.985204 21620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:35813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab
--principal=kudu/127.10.238.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.238.194
--webserver_port=0
--tserver_master_addrs=127.10.238.254:45429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.194
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:55:28.985483 21620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:55:28.985736 21620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 10:55:28.991065 21620 server_base.cc:1034] running on GCE node
W20260929 10:55:28.991019 21628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:28.991010 21626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:28.991010 21625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:55:28.991452 21620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:55:28.991710 21620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:55:28.992854 21620 hybrid_clock.cc:648] HybridClock initialized: now 1790679328992839 us; error 34 us; skew 500 ppm
Sep 29 10:55:28 dist-test-slave-8nrt krb5kdc[21379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679328, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:55:28.995718 21620 init.cc:377] Logged in from keytab as kudu/127.10.238.194@KRBTEST.COM (short username kudu)
I20260929 10:55:28.996197 21620 webserver.cc:460] Webserver started at http://127.10.238.194:36591/ using document root <none> and password file <none>
I20260929 10:55:28.996464 21620 fs_manager.cc:359] Metadata directory not provided
I20260929 10:55:28.996510 21620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:55:28.996650 21620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:55:28.997813 21620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/ts-1/data/instance:
uuid: "a800d8504bd64dfcaacee122c5172736"
format_stamp: "Formatted at 2026-09-29 10:55:28 on dist-test-slave-8nrt"
server_key: "e83dcde5c146a750dec35ffdf9ff60ad"
server_key_iv: "7b068a48a984023601bba7c4f5c18c56"
server_key_version: "encryptionkey@0"
I20260929 10:55:28.998176 21620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/ts-1/wal/instance:
uuid: "a800d8504bd64dfcaacee122c5172736"
format_stamp: "Formatted at 2026-09-29 10:55:28 on dist-test-slave-8nrt"
server_key: "e83dcde5c146a750dec35ffdf9ff60ad"
server_key_iv: "7b068a48a984023601bba7c4f5c18c56"
server_key_version: "encryptionkey@0"
I20260929 10:55:28.999512 21620 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 10:55:29.000267 21635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:29.000469 21620 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:55:29.000560 21620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/ts-1/wal
uuid: "a800d8504bd64dfcaacee122c5172736"
format_stamp: "Formatted at 2026-09-29 10:55:28 on dist-test-slave-8nrt"
server_key: "e83dcde5c146a750dec35ffdf9ff60ad"
server_key_iv: "7b068a48a984023601bba7c4f5c18c56"
server_key_version: "encryptionkey@0"
I20260929 10:55:29.000654 21620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:55:29.022583 21620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:55:29.025054 21620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:55:29.025169 21620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:55:29.025484 21620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:55:29.025825 21620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:55:29.025861 21620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:29.025883 21620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:55:29.025915 21620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:29.031795 21620 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.194:43583
I20260929 10:55:29.031836 21748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.194:43583 every 8 connection(s)
I20260929 10:55:29.032162 21620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
I20260929 10:55:29.035534 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 21620
I20260929 10:55:29.035642 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/ts-1/wal/instance
I20260929 10:55:29.035866 11195 external_mini_cluster.cc:1428] Setting key c217e7cfeb6c8d7af4e975d7d3d54a87
Sep 29 10:55:29 dist-test-slave-8nrt krb5kdc[21379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679328, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.194@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.238.195@KRBTEST.COM; defaulting to no policy
I20260929 10:55:29.043953 21464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:29.032970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:39225 (local address 127.10.238.254:45429)
0929 10:55:29.033153 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:29.033155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:29.033320 (+ 165us) server_negotiation.cc:407] Connection header received
0929 10:55:29.034098 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:29.034100 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:29.034114 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:29.034183 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:29.035581 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:29.036367 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:29.037588 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:29.037911 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:29.037996 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:29.040609 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:29.040610 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:29.040611 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:29.040621 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:29.042203 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:29.042854 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:29.042856 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:29.042857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:29.042932 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:29.043227 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:29.043228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:29.043228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:29.043362 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:29.043451 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:29.043756 (+ 305us) server_negotiation.cc:299] Negotiation successful
0929 10:55:29.043852 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260929 10:55:29.044183 21751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:29.032929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:45429 (local address 127.10.238.194:39225)
0929 10:55:29.033129 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:29.033201 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:29.033987 (+ 786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:29.034254 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:29.034259 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:29.035093 (+ 834us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:29.035415 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:29.035423 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:29.036477 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:29.036478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:29.037505 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:29.037508 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:29.038025 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:29.038027 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:29.038209 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:29.038815 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:29.038826 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:29.040508 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:55:29.042459 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:29.042473 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:29.042475 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:29.042750 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:29.043011 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:29.043012 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:29.043012 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:29.043148 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:29.043439 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:55:29.043447 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:55:29.043604 (+ 157us) client_negotiation.cc:769] Sending connection context
0929 10:55:29.043721 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 10:55:29.043901 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":79,"threads_started":1}
I20260929 10:55:29.044732 21749 heartbeater.cc:347] Connected to a master server at 127.10.238.254:45429
I20260929 10:55:29.044828 21749 heartbeater.cc:464] Registering TS with master...
I20260929 10:55:29.045054 21749 heartbeater.cc:511] Master 127.10.238.254:45429 requested a full tablet report, sending...
I20260929 10:55:29.045576 21424 ts_manager.cc:194] Registered new tserver with Master: a800d8504bd64dfcaacee122c5172736 (127.10.238.194:43583)
I20260929 10:55:29.046022 21424 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.194@KRBTEST.COM'} at 127.10.238.194:39225
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab.
Entry for principal kudu/127.10.238.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab.
I20260929 10:55:29.062490 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.195:0
--local_ip_for_outbound_sockets=127.10.238.195
--webserver_interface=127.10.238.195
--webserver_port=0
--tserver_master_addrs=127.10.238.254:45429
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab
--principal=kudu/127.10.238.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:35813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:55:29.145565 21754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:55:29.145803 21754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:55:29.145828 21754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:55:29.147119 21754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:55:29.147166 21754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:55:29.147194 21754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.195
I20260929 10:55:29.148979 21754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:35813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab
--principal=kudu/127.10.238.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.238.195
--webserver_port=0
--tserver_master_addrs=127.10.238.254:45429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.195
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:55:29.149248 21754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:55:29.149561 21754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:55:29.154645 21760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:29.154717 21759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:55:29.154706 21762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:55:29.154824 21754 server_base.cc:1034] running on GCE node
I20260929 10:55:29.155032 21754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:55:29.155241 21754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:55:29.156401 21754 hybrid_clock.cc:648] HybridClock initialized: now 1790679329156383 us; error 41 us; skew 500 ppm
Sep 29 10:55:29 dist-test-slave-8nrt krb5kdc[21379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679329, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:55:29.159078 21754 init.cc:377] Logged in from keytab as kudu/127.10.238.195@KRBTEST.COM (short username kudu)
I20260929 10:55:29.159691 21754 webserver.cc:460] Webserver started at http://127.10.238.195:36377/ using document root <none> and password file <none>
I20260929 10:55:29.159945 21754 fs_manager.cc:359] Metadata directory not provided
I20260929 10:55:29.159991 21754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:55:29.160142 21754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:55:29.161202 21754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/ts-2/data/instance:
uuid: "a1ec70068a5845edb022eb7757bd4f46"
format_stamp: "Formatted at 2026-09-29 10:55:29 on dist-test-slave-8nrt"
server_key: "e5637794b7afef7eb975acdbc2d7e9c9"
server_key_iv: "c110788844e3d2afe70fefbd6473aaaf"
server_key_version: "encryptionkey@0"
I20260929 10:55:29.161645 21754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/ts-2/wal/instance:
uuid: "a1ec70068a5845edb022eb7757bd4f46"
format_stamp: "Formatted at 2026-09-29 10:55:29 on dist-test-slave-8nrt"
server_key: "e5637794b7afef7eb975acdbc2d7e9c9"
server_key_iv: "c110788844e3d2afe70fefbd6473aaaf"
server_key_version: "encryptionkey@0"
I20260929 10:55:29.163110 21754 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:55:29.163995 21769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:29.164235 21754 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:55:29.164341 21754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/ts-2/wal
uuid: "a1ec70068a5845edb022eb7757bd4f46"
format_stamp: "Formatted at 2026-09-29 10:55:29 on dist-test-slave-8nrt"
server_key: "e5637794b7afef7eb975acdbc2d7e9c9"
server_key_iv: "c110788844e3d2afe70fefbd6473aaaf"
server_key_version: "encryptionkey@0"
I20260929 10:55:29.164425 21754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:55:29.193686 21754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:55:29.195917 21754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:55:29.196033 21754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:55:29.196228 21754 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:55:29.196620 21754 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:55:29.196656 21754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:29.196681 21754 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:55:29.196730 21754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:55:29.203164 21754 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.195:37645
I20260929 10:55:29.203207 21882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.195:37645 every 8 connection(s)
I20260929 10:55:29.203593 21754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
I20260929 10:55:29.207971 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 21754
I20260929 10:55:29.208094 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/ts-2/wal/instance
I20260929 10:55:29.208278 11195 external_mini_cluster.cc:1428] Setting key cf495dbe9d85c554935f86f1e8fdc3e3
Sep 29 10:55:29 dist-test-slave-8nrt krb5kdc[21379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679329, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.195@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:55:29.213238 21464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:29.204296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:38479 (local address 127.10.238.254:45429)
0929 10:55:29.204342 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:29.204344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:29.204579 (+ 235us) server_negotiation.cc:407] Connection header received
0929 10:55:29.205133 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:29.205134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:29.205144 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:29.205196 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:29.206141 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:29.206766 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:29.207752 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:29.207946 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:29.208013 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:29.210529 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:29.210530 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:29.210530 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:29.210537 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:29.212010 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:29.212362 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:29.212363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:29.212363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:29.212416 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:29.212694 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:29.212694 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:29.212695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:29.212821 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:29.212869 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:29.213108 (+ 239us) server_negotiation.cc:299] Negotiation successful
0929 10:55:29.213171 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 10:55:29.213472 21885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:29.204266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:45429 (local address 127.10.238.195:38479)
0929 10:55:29.204421 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:29.204458 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:29.205042 (+ 584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:29.205266 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:29.205269 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:29.205854 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:55:29.206057 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:29.206061 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:29.206858 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:29.206859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:29.207666 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:29.207668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:29.208042 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:29.208044 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:29.208214 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:29.208930 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:55:29.208940 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:55:29.210449 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:55:29.212079 (+ 1630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:29.212084 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:29.212086 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:29.212297 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:29.212473 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:55:29.212474 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:55:29.212474 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:55:29.212597 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:55:29.212870 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:55:29.212874 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:55:29.212996 (+ 122us) client_negotiation.cc:769] Sending connection context
0929 10:55:29.213077 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 10:55:29.213201 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":91,"threads_started":1}
I20260929 10:55:29.214001 21883 heartbeater.cc:347] Connected to a master server at 127.10.238.254:45429
I20260929 10:55:29.214100 21883 heartbeater.cc:464] Registering TS with master...
I20260929 10:55:29.214272 21883 heartbeater.cc:511] Master 127.10.238.254:45429 requested a full tablet report, sending...
I20260929 10:55:29.214653 21424 ts_manager.cc:194] Registered new tserver with Master: a1ec70068a5845edb022eb7757bd4f46 (127.10.238.195:37645)
I20260929 10:55:29.215044 21424 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.195@KRBTEST.COM'} at 127.10.238.195:38479
I20260929 10:55:29.220055 11195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:55:29.227737 21464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:29.221464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48002 (local address 127.10.238.254:45429)
0929 10:55:29.221544 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:29.221546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:29.221667 (+ 121us) server_negotiation.cc:407] Connection header received
0929 10:55:29.221739 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:29.221740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:29.221751 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:29.221824 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:55:29.222512 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:29.223069 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:29.223804 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:29.224007 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:29.224081 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:29.225249 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:55:29.225250 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:55:29.225251 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:55:29.225273 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:55:29.226531 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:29.226897 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:29.226898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:29.226899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:29.226960 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:55:29.227172 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:55:29.227173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:55:29.227174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:55:29.227337 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:55:29.227441 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:29.227551 (+ 110us) server_negotiation.cc:299] Negotiation successful
0929 10:55:29.227633 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 10:55:29.229305 21424 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48002:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:55:29.230953 21424 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:55:29.238046 21899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:29.233795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56880 (local address 127.10.238.195:37645)
0929 10:55:29.233979 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:29.233983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:29.234228 (+ 245us) server_negotiation.cc:407] Connection header received
0929 10:55:29.234277 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:29.234279 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:29.234297 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:29.234369 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:55:29.234602 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:29.235302 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:29.236979 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:29.237576 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:29.237677 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:29.237741 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:29.237909 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 10:55:29.237950 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":108,"threads_started":1}
I20260929 10:55:29.238096 21894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:29.233395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:41053 (local address 127.0.0.1:49914)
0929 10:55:29.233662 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:29.233712 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:29.233784 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:29.234194 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:29.234197 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:29.234200 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:55:29.234301 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:29.234303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:29.235611 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:29.235611 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:29.236875 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:29.236878 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:29.237813 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:29.237814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:29.237883 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:29.237889 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 10:55:29.237940 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 10:55:29.237995 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":58,"thread_start_us":54,"threads_started":1}
I20260929 10:55:29.238186 21897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:29.233596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52922 (local address 127.10.238.194:43583)
0929 10:55:29.233769 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:29.233775 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:29.234092 (+ 317us) server_negotiation.cc:407] Connection header received
0929 10:55:29.234360 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:29.234363 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:29.234379 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:29.234429 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:55:29.234673 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:29.235707 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:29.236847 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:29.237625 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:29.237703 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:29.237772 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:29.238039 (+ 267us) server_negotiation.cc:299] Negotiation successful
0929 10:55:29.238082 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":99,"threads_started":1}
I20260929 10:55:29.238459 21898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:29.233653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49914 (local address 127.10.238.193:41053)
0929 10:55:29.233925 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:29.233929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:29.233978 (+ 49us) server_negotiation.cc:407] Connection header received
0929 10:55:29.234012 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:29.234014 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:29.234028 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:29.234118 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:55:29.234390 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:29.235502 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:29.236964 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:29.237681 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:29.237740 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:29.237785 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:29.238303 (+ 518us) server_negotiation.cc:299] Negotiation successful
0929 10:55:29.238365 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":135,"threads_started":1}
I20260929 10:55:29.238692 21896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:29.233398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.195:37645 (local address 127.0.0.1:56880)
0929 10:55:29.234138 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:29.234147 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:29.234195 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:29.234424 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:29.234425 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:29.234427 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:55:29.234482 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:29.234485 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:29.235416 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:29.235417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:29.236875 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:29.236878 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:29.237675 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:29.237676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:29.237765 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:29.237779 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 10:55:29.237890 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 10:55:29.237950 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"mutex_wait_us":62,"thread_start_us":93,"threads_started":1}
I20260929 10:55:29.239074 21895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:29.233395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.194:43583 (local address 127.0.0.1:52922)
0929 10:55:29.233999 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:29.234008 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:29.234253 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:29.234532 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:29.234533 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:29.234535 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:55:29.234576 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:29.234577 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:29.235819 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:29.235821 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:29.236755 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:29.236758 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:29.237812 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:29.237813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:29.237883 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:29.237891 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 10:55:29.237940 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 10:55:29.237995 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"mutex_wait_us":13,"spinlock_wait_cycles":768,"thread_start_us":107,"threads_started":1}
I20260929 10:55:29.239228 21549 tablet_service.cc:1467] Processing CreateTablet for tablet 40160db127494ce5a376eb4b25b4cc10 (DEFAULT_TABLE table=test-table [id=42d362fb4ebe4dc0bb6fbb86903faffa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:55:29.239420 21817 tablet_service.cc:1467] Processing CreateTablet for tablet 40160db127494ce5a376eb4b25b4cc10 (DEFAULT_TABLE table=test-table [id=42d362fb4ebe4dc0bb6fbb86903faffa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:55:29.239595 21549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40160db127494ce5a376eb4b25b4cc10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:55:29.239756 21817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40160db127494ce5a376eb4b25b4cc10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:55:29.242535 21900 tablet_bootstrap.cc:492] T 40160db127494ce5a376eb4b25b4cc10 P a1ec70068a5845edb022eb7757bd4f46: Bootstrap starting.
I20260929 10:55:29.242669 21901 tablet_bootstrap.cc:492] T 40160db127494ce5a376eb4b25b4cc10 P d616379489274a46928923186f4248ed: Bootstrap starting.
I20260929 10:55:29.243270 21901 tablet_bootstrap.cc:654] T 40160db127494ce5a376eb4b25b4cc10 P d616379489274a46928923186f4248ed: Neither blocks nor log segments found. Creating new log.
I20260929 10:55:29.243273 21900 tablet_bootstrap.cc:654] T 40160db127494ce5a376eb4b25b4cc10 P a1ec70068a5845edb022eb7757bd4f46: Neither blocks nor log segments found. Creating new log.
I20260929 10:55:29.243299 21683 tablet_service.cc:1467] Processing CreateTablet for tablet 40160db127494ce5a376eb4b25b4cc10 (DEFAULT_TABLE table=test-table [id=42d362fb4ebe4dc0bb6fbb86903faffa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:55:29.243650 21683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40160db127494ce5a376eb4b25b4cc10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:55:29.243901 21901 log.cc:824] T 40160db127494ce5a376eb4b25b4cc10 P d616379489274a46928923186f4248ed: Log is configured to *not* fsync() on all Append() calls
I20260929 10:55:29.243904 21900 log.cc:824] T 40160db127494ce5a376eb4b25b4cc10 P a1ec70068a5845edb022eb7757bd4f46: Log is configured to *not* fsync() on all Append() calls
I20260929 10:55:29.244680 21901 tablet_bootstrap.cc:492] T 40160db127494ce5a376eb4b25b4cc10 P d616379489274a46928923186f4248ed: No bootstrap required, opened a new log
I20260929 10:55:29.244765 21901 ts_tablet_manager.cc:1397] T 40160db127494ce5a376eb4b25b4cc10 P d616379489274a46928923186f4248ed: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 10:55:29.244939 21900 tablet_bootstrap.cc:492] T 40160db127494ce5a376eb4b25b4cc10 P a1ec70068a5845edb022eb7757bd4f46: No bootstrap required, opened a new log
I20260929 10:55:29.245368 21900 ts_tablet_manager.cc:1397] T 40160db127494ce5a376eb4b25b4cc10 P a1ec70068a5845edb022eb7757bd4f46: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 10:55:29.246345 21901 raft_consensus.cc:348] T 40160db127494ce5a376eb4b25b4cc10 P d616379489274a46928923186f4248ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a800d8504bd64dfcaacee122c5172736" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 43583 } } peers { permanent_uuid: "a1ec70068a5845edb022eb7757bd4f46" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 37645 } } peers { permanent_uuid: "d616379489274a46928923186f4248ed" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 41053 } }
I20260929 10:55:29.246500 21901 raft_consensus.cc:374] T 40160db127494ce5a376eb4b25b4cc10 P d616379489274a46928923186f4248ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:55:29.246542 21901 raft_consensus.cc:729] T 40160db127494ce5a376eb4b25b4cc10 P d616379489274a46928923186f4248ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d616379489274a46928923186f4248ed, State: Initialized, Role: FOLLOWER
I20260929 10:55:29.246667 21901 consensus_queue.cc:260] T 40160db127494ce5a376eb4b25b4cc10 P d616379489274a46928923186f4248ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a800d8504bd64dfcaacee122c5172736" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 43583 } } peers { permanent_uuid: "a1ec70068a5845edb022eb7757bd4f46" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 37645 } } peers { permanent_uuid: "d616379489274a46928923186f4248ed" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 41053 } }
I20260929 10:55:29.246981 21904 tablet_bootstrap.cc:492] T 40160db127494ce5a376eb4b25b4cc10 P a800d8504bd64dfcaacee122c5172736: Bootstrap starting.
I20260929 10:55:29.247155 21901 ts_tablet_manager.cc:1428] T 40160db127494ce5a376eb4b25b4cc10 P d616379489274a46928923186f4248ed: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 10:55:29.247177 21615 heartbeater.cc:503] Master 127.10.238.254:45429 was elected leader, sending a full tablet report...
I20260929 10:55:29.247668 21904 tablet_bootstrap.cc:654] T 40160db127494ce5a376eb4b25b4cc10 P a800d8504bd64dfcaacee122c5172736: Neither blocks nor log segments found. Creating new log.
I20260929 10:55:29.247786 21900 raft_consensus.cc:348] T 40160db127494ce5a376eb4b25b4cc10 P a1ec70068a5845edb022eb7757bd4f46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a800d8504bd64dfcaacee122c5172736" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 43583 } } peers { permanent_uuid: "a1ec70068a5845edb022eb7757bd4f46" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 37645 } } peers { permanent_uuid: "d616379489274a46928923186f4248ed" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 41053 } }
I20260929 10:55:29.247958 21900 raft_consensus.cc:374] T 40160db127494ce5a376eb4b25b4cc10 P a1ec70068a5845edb022eb7757bd4f46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:55:29.248001 21900 raft_consensus.cc:729] T 40160db127494ce5a376eb4b25b4cc10 P a1ec70068a5845edb022eb7757bd4f46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1ec70068a5845edb022eb7757bd4f46, State: Initialized, Role: FOLLOWER
I20260929 10:55:29.248139 21900 consensus_queue.cc:260] T 40160db127494ce5a376eb4b25b4cc10 P a1ec70068a5845edb022eb7757bd4f46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a800d8504bd64dfcaacee122c5172736" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 43583 } } peers { permanent_uuid: "a1ec70068a5845edb022eb7757bd4f46" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 37645 } } peers { permanent_uuid: "d616379489274a46928923186f4248ed" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 41053 } }
I20260929 10:55:29.248323 21904 log.cc:824] T 40160db127494ce5a376eb4b25b4cc10 P a800d8504bd64dfcaacee122c5172736: Log is configured to *not* fsync() on all Append() calls
I20260929 10:55:29.248405 21900 ts_tablet_manager.cc:1428] T 40160db127494ce5a376eb4b25b4cc10 P a1ec70068a5845edb022eb7757bd4f46: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 10:55:29.248512 21883 heartbeater.cc:503] Master 127.10.238.254:45429 was elected leader, sending a full tablet report...
I20260929 10:55:29.249122 21904 tablet_bootstrap.cc:492] T 40160db127494ce5a376eb4b25b4cc10 P a800d8504bd64dfcaacee122c5172736: No bootstrap required, opened a new log
I20260929 10:55:29.249202 21904 ts_tablet_manager.cc:1397] T 40160db127494ce5a376eb4b25b4cc10 P a800d8504bd64dfcaacee122c5172736: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 10:55:29.250639 21904 raft_consensus.cc:348] T 40160db127494ce5a376eb4b25b4cc10 P a800d8504bd64dfcaacee122c5172736 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a800d8504bd64dfcaacee122c5172736" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 43583 } } peers { permanent_uuid: "a1ec70068a5845edb022eb7757bd4f46" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 37645 } } peers { permanent_uuid: "d616379489274a46928923186f4248ed" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 41053 } }
I20260929 10:55:29.250789 21904 raft_consensus.cc:374] T 40160db127494ce5a376eb4b25b4cc10 P a800d8504bd64dfcaacee122c5172736 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:55:29.250834 21904 raft_consensus.cc:729] T 40160db127494ce5a376eb4b25b4cc10 P a800d8504bd64dfcaacee122c5172736 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a800d8504bd64dfcaacee122c5172736, State: Initialized, Role: FOLLOWER
I20260929 10:55:29.250941 21904 consensus_queue.cc:260] T 40160db127494ce5a376eb4b25b4cc10 P a800d8504bd64dfcaacee122c5172736 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a800d8504bd64dfcaacee122c5172736" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 43583 } } peers { permanent_uuid: "a1ec70068a5845edb022eb7757bd4f46" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 37645 } } peers { permanent_uuid: "d616379489274a46928923186f4248ed" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 41053 } }
I20260929 10:55:29.251195 21904 ts_tablet_manager.cc:1428] T 40160db127494ce5a376eb4b25b4cc10 P a800d8504bd64dfcaacee122c5172736: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 10:55:29.251283 21749 heartbeater.cc:503] Master 127.10.238.254:45429 was elected leader, sending a full tablet report...
W20260929 10:55:29.282752 21750 tablet.cc:2369] T 40160db127494ce5a376eb4b25b4cc10 P a800d8504bd64dfcaacee122c5172736: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 10:55:29.366065 21616 tablet.cc:2369] T 40160db127494ce5a376eb4b25b4cc10 P d616379489274a46928923186f4248ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:55:29.417109 21905 raft_consensus.cc:482] T 40160db127494ce5a376eb4b25b4cc10 P d616379489274a46928923186f4248ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:55:29.417229 21905 raft_consensus.cc:504] T 40160db127494ce5a376eb4b25b4cc10 P d616379489274a46928923186f4248ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a800d8504bd64dfcaacee122c5172736" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 43583 } } peers { permanent_uuid: "a1ec70068a5845edb022eb7757bd4f46" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 37645 } } peers { permanent_uuid: "d616379489274a46928923186f4248ed" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 41053 } }
I20260929 10:55:29.417610 21905 leader_election.cc:288] T 40160db127494ce5a376eb4b25b4cc10 P d616379489274a46928923186f4248ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a800d8504bd64dfcaacee122c5172736 (127.10.238.194:43583), a1ec70068a5845edb022eb7757bd4f46 (127.10.238.195:37645)
I20260929 10:55:29.421772 21899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:29.417941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:35049 (local address 127.10.238.195:37645)
0929 10:55:29.417983 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:29.417985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:29.418045 (+ 60us) server_negotiation.cc:407] Connection header received
0929 10:55:29.418124 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:29.418125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:29.418134 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:29.418190 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:55:29.418412 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:29.419159 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:29.420761 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:29.421407 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:29.421485 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:29.421494 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:29.421658 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 10:55:29.421696 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 10:55:29.421772 21897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:29.417893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:59471 (local address 127.10.238.194:43583)
0929 10:55:29.417944 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:29.417946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:29.418046 (+ 100us) server_negotiation.cc:407] Connection header received
0929 10:55:29.418108 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:29.418110 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:29.418120 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:29.418190 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:55:29.418410 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:29.419173 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:29.420760 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:29.421406 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:29.421485 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:29.421494 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:29.421668 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 10:55:29.421702 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 10:55:29.421825 21910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:29.417802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.194:43583 (local address 127.10.238.193:59471)
0929 10:55:29.417966 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:29.417975 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:29.418046 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:29.418212 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:29.418213 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:29.418215 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:55:29.418320 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:29.418323 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:29.419261 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:29.419262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:29.420668 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:29.420671 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:29.421489 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:29.421490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:29.421573 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:29.421580 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 10:55:29.421642 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 10:55:29.421698 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":129,"thread_start_us":71,"threads_started":1}
I20260929 10:55:29.421829 21909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:29.417802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.195:37645 (local address 127.10.238.193:35049)
0929 10:55:29.417962 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:29.417975 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:55:29.418046 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:55:29.418212 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:55:29.418213 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:55:29.418216 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:55:29.418331 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:29.418333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:29.419250 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:29.419250 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:29.420668 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:55:29.420671 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:55:29.421491 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:55:29.421492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:55:29.421573 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:29.421581 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 10:55:29.421642 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 10:55:29.421698 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":78,"thread_start_us":92,"threads_started":1}
I20260929 10:55:29.422256 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40160db127494ce5a376eb4b25b4cc10" candidate_uuid: "d616379489274a46928923186f4248ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a800d8504bd64dfcaacee122c5172736" is_pre_election: true
I20260929 10:55:29.422458 21703 raft_consensus.cc:2454] T 40160db127494ce5a376eb4b25b4cc10 P a800d8504bd64dfcaacee122c5172736 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d616379489274a46928923186f4248ed in term 0.
I20260929 10:55:29.422482 21837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40160db127494ce5a376eb4b25b4cc10" candidate_uuid: "d616379489274a46928923186f4248ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1ec70068a5845edb022eb7757bd4f46" is_pre_election: true
I20260929 10:55:29.422664 21837 raft_consensus.cc:2454] T 40160db127494ce5a376eb4b25b4cc10 P a1ec70068a5845edb022eb7757bd4f46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d616379489274a46928923186f4248ed in term 0.
I20260929 10:55:29.422688 21503 leader_election.cc:302] T 40160db127494ce5a376eb4b25b4cc10 P d616379489274a46928923186f4248ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a800d8504bd64dfcaacee122c5172736, d616379489274a46928923186f4248ed; no voters:
I20260929 10:55:29.422848 21905 raft_consensus.cc:2781] T 40160db127494ce5a376eb4b25b4cc10 P d616379489274a46928923186f4248ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:55:29.422923 21905 raft_consensus.cc:482] T 40160db127494ce5a376eb4b25b4cc10 P d616379489274a46928923186f4248ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:55:29.422952 21905 raft_consensus.cc:3037] T 40160db127494ce5a376eb4b25b4cc10 P d616379489274a46928923186f4248ed [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:55:29.423753 21905 raft_consensus.cc:504] T 40160db127494ce5a376eb4b25b4cc10 P d616379489274a46928923186f4248ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a800d8504bd64dfcaacee122c5172736" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 43583 } } peers { permanent_uuid: "a1ec70068a5845edb022eb7757bd4f46" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 37645 } } peers { permanent_uuid: "d616379489274a46928923186f4248ed" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 41053 } }
I20260929 10:55:29.423907 21905 leader_election.cc:288] T 40160db127494ce5a376eb4b25b4cc10 P d616379489274a46928923186f4248ed [CANDIDATE]: Term 1 election: Requested vote from peers a800d8504bd64dfcaacee122c5172736 (127.10.238.194:43583), a1ec70068a5845edb022eb7757bd4f46 (127.10.238.195:37645)
I20260929 10:55:29.424091 21837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40160db127494ce5a376eb4b25b4cc10" candidate_uuid: "d616379489274a46928923186f4248ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1ec70068a5845edb022eb7757bd4f46"
I20260929 10:55:29.424177 21837 raft_consensus.cc:3037] T 40160db127494ce5a376eb4b25b4cc10 P a1ec70068a5845edb022eb7757bd4f46 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:55:29.424908 21837 raft_consensus.cc:2454] T 40160db127494ce5a376eb4b25b4cc10 P a1ec70068a5845edb022eb7757bd4f46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d616379489274a46928923186f4248ed in term 1.
I20260929 10:55:29.425041 21703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40160db127494ce5a376eb4b25b4cc10" candidate_uuid: "d616379489274a46928923186f4248ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a800d8504bd64dfcaacee122c5172736"
I20260929 10:55:29.425108 21502 leader_election.cc:302] T 40160db127494ce5a376eb4b25b4cc10 P d616379489274a46928923186f4248ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1ec70068a5845edb022eb7757bd4f46, d616379489274a46928923186f4248ed; no voters:
I20260929 10:55:29.425120 21703 raft_consensus.cc:3037] T 40160db127494ce5a376eb4b25b4cc10 P a800d8504bd64dfcaacee122c5172736 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:55:29.425884 21703 raft_consensus.cc:2454] T 40160db127494ce5a376eb4b25b4cc10 P a800d8504bd64dfcaacee122c5172736 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d616379489274a46928923186f4248ed in term 1.
I20260929 10:55:29.426007 21905 raft_consensus.cc:2781] T 40160db127494ce5a376eb4b25b4cc10 P d616379489274a46928923186f4248ed [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:55:29.426170 21905 raft_consensus.cc:686] T 40160db127494ce5a376eb4b25b4cc10 P d616379489274a46928923186f4248ed [term 1 LEADER]: Becoming Leader. State: Replica: d616379489274a46928923186f4248ed, State: Running, Role: LEADER
I20260929 10:55:29.426290 21905 consensus_queue.cc:237] T 40160db127494ce5a376eb4b25b4cc10 P d616379489274a46928923186f4248ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a800d8504bd64dfcaacee122c5172736" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 43583 } } peers { permanent_uuid: "a1ec70068a5845edb022eb7757bd4f46" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 37645 } } peers { permanent_uuid: "d616379489274a46928923186f4248ed" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 41053 } }
I20260929 10:55:29.427172 21424 catalog_manager.cc:5495] T 40160db127494ce5a376eb4b25b4cc10 P d616379489274a46928923186f4248ed reported cstate change: term changed from 0 to 1, leader changed from <none> to d616379489274a46928923186f4248ed (127.10.238.193). New cstate: current_term: 1 leader_uuid: "d616379489274a46928923186f4248ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a800d8504bd64dfcaacee122c5172736" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 43583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1ec70068a5845edb022eb7757bd4f46" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 37645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d616379489274a46928923186f4248ed" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 41053 } health_report { overall_health: HEALTHY } } }
W20260929 10:55:29.454191 21884 tablet.cc:2369] T 40160db127494ce5a376eb4b25b4cc10 P a1ec70068a5845edb022eb7757bd4f46: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:55:29.463629 21898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:29.461367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49924 (local address 127.10.238.193:41053)
0929 10:55:29.461422 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 10:55:29.461424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:55:29.461431 (+ 7us) server_negotiation.cc:407] Connection header received
0929 10:55:29.461485 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:55:29.461486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:55:29.461505 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:55:29.461557 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:55:29.461683 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:29.462157 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:29.462884 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:55:29.463091 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:55:29.463143 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:55:29.463296 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:55:29.463401 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:55:29.463462 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 10:55:29.463496 (+ 34us) server_negotiation.cc:299] Negotiation successful
0929 10:55:29.463514 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 10:55:29.465579 21703 raft_consensus.cc:1261] T 40160db127494ce5a376eb4b25b4cc10 P a800d8504bd64dfcaacee122c5172736 [term 1 FOLLOWER]: Refusing update from remote peer d616379489274a46928923186f4248ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:55:29.465579 21837 raft_consensus.cc:1261] T 40160db127494ce5a376eb4b25b4cc10 P a1ec70068a5845edb022eb7757bd4f46 [term 1 FOLLOWER]: Refusing update from remote peer d616379489274a46928923186f4248ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:55:29.465991 21905 consensus_queue.cc:1035] T 40160db127494ce5a376eb4b25b4cc10 P d616379489274a46928923186f4248ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1ec70068a5845edb022eb7757bd4f46" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:55:29.466193 21911 consensus_queue.cc:1035] T 40160db127494ce5a376eb4b25b4cc10 P d616379489274a46928923186f4248ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "a800d8504bd64dfcaacee122c5172736" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 43583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:55:29.468430 21914 mvcc.cc:205] Tried to move back new op lower bound from 7334622533488099328 to 7334622533330825216. Current Snapshot: MvccSnapshot[applied={T|T < 7334622533488099328}]
I20260929 10:55:29.468834 21915 mvcc.cc:205] Tried to move back new op lower bound from 7334622533488099328 to 7334622533330825216. Current Snapshot: MvccSnapshot[applied={T|T < 7334622533488099328}]
I20260929 10:55:29.469411 21916 mvcc.cc:205] Tried to move back new op lower bound from 7334622533488099328 to 7334622533330825216. Current Snapshot: MvccSnapshot[applied={T|T < 7334622533488099328 or (T in {7334622533488099328})}]
I20260929 10:55:29.471561 21423 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48002:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:55:29.471643 21423 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48002:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:55:29.472426 21423 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 59815ec992df4f5eb6b399f54a5888fe: Sending DeleteTablet for 3 replicas of tablet 40160db127494ce5a376eb4b25b4cc10
I20260929 10:55:29.472836 21549 tablet_service.cc:1514] Processing DeleteTablet for tablet 40160db127494ce5a376eb4b25b4cc10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:55:29 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:49914
I20260929 10:55:29.472869 21683 tablet_service.cc:1514] Processing DeleteTablet for tablet 40160db127494ce5a376eb4b25b4cc10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:55:29 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:52922
I20260929 10:55:29.473047 21817 tablet_service.cc:1514] Processing DeleteTablet for tablet 40160db127494ce5a376eb4b25b4cc10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:55:29 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:56880
I20260929 10:55:29.473191 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 21486
I20260929 10:55:29.476786 21926 tablet_replica.cc:330] T 40160db127494ce5a376eb4b25b4cc10 P a1ec70068a5845edb022eb7757bd4f46: stopping tablet replica
I20260929 10:55:29.476922 21926 raft_consensus.cc:2229] T 40160db127494ce5a376eb4b25b4cc10 P a1ec70068a5845edb022eb7757bd4f46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:55:29.477027 21925 tablet_replica.cc:330] T 40160db127494ce5a376eb4b25b4cc10 P a800d8504bd64dfcaacee122c5172736: stopping tablet replica
I20260929 10:55:29.477058 21926 raft_consensus.cc:2258] T 40160db127494ce5a376eb4b25b4cc10 P a1ec70068a5845edb022eb7757bd4f46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:55:29.477134 21925 raft_consensus.cc:2229] T 40160db127494ce5a376eb4b25b4cc10 P a800d8504bd64dfcaacee122c5172736 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:55:29.477239 21925 raft_consensus.cc:2258] T 40160db127494ce5a376eb4b25b4cc10 P a800d8504bd64dfcaacee122c5172736 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:55:29.477617 21926 ts_tablet_manager.cc:1905] T 40160db127494ce5a376eb4b25b4cc10 P a1ec70068a5845edb022eb7757bd4f46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:55:29.477684 21925 ts_tablet_manager.cc:1905] T 40160db127494ce5a376eb4b25b4cc10 P a800d8504bd64dfcaacee122c5172736: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 10:55:29.479018 21411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.238.193:41053 (error 108)
I20260929 10:55:29.479351 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 21620
I20260929 10:55:29.479490 21925 ts_tablet_manager.cc:1918] T 40160db127494ce5a376eb4b25b4cc10 P a800d8504bd64dfcaacee122c5172736: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:55:29.479557 21925 log.cc:1196] T 40160db127494ce5a376eb4b25b4cc10 P a800d8504bd64dfcaacee122c5172736: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/ts-1/wal/wals/40160db127494ce5a376eb4b25b4cc10
I20260929 10:55:29.480075 21895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:29.479429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:41053 (local address 127.0.0.1:49934)
0929 10:55:29.479987 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:29.480032 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.193:41053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":537}
W20260929 10:55:29.480333 21411 catalog_manager.cc:4584] TS d616379489274a46928923186f4248ed (127.10.238.193:41053): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 40160db127494ce5a376eb4b25b4cc10: Network error: Client connection negotiation failed: client connection to 127.10.238.193:41053: connect: Connection refused (error 111)
I20260929 10:55:29.482136 21926 ts_tablet_manager.cc:1918] T 40160db127494ce5a376eb4b25b4cc10 P a1ec70068a5845edb022eb7757bd4f46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:55:29.482216 21926 log.cc:1196] T 40160db127494ce5a376eb4b25b4cc10 P a1ec70068a5845edb022eb7757bd4f46: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790679274510869-11195-0/minicluster-data/ts-2/wal/wals/40160db127494ce5a376eb4b25b4cc10
I20260929 10:55:29.482486 21926 ts_tablet_manager.cc:1939] T 40160db127494ce5a376eb4b25b4cc10 P a1ec70068a5845edb022eb7757bd4f46: Deleting consensus metadata
I20260929 10:55:29.482916 21408 catalog_manager.cc:4841] TS a1ec70068a5845edb022eb7757bd4f46 (127.10.238.195:37645): tablet 40160db127494ce5a376eb4b25b4cc10 (table test-table [id=42d362fb4ebe4dc0bb6fbb86903faffa]) successfully deleted
I20260929 10:55:29.484467 21895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:55:29.484347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.194:43583 (local address 127.0.0.1:52924)
0929 10:55:29.484402 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0929 10:55:29.484434 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.194:43583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260929 10:55:29.484822 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 21754
I20260929 10:55:29.490228 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 21392
2026-09-29T10:55:29Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2133 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:55:29 dist-test-slave-8nrt krb5kdc[21930](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:55:29 dist-test-slave-8nrt krb5kdc[21930](info): set up 2 sockets
Sep 29 10:55:29 dist-test-slave-8nrt krb5kdc[21930](info): commencing operation
krb5kdc: starting...
W20260929 10:55:30.526304 11195 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Sep 29 10:55:30 dist-test-slave-8nrt krb5kdc[21930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:55:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:55:30Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.10.238.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.10.238.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.10.238.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/krb5kdc/rangeradmin_127.10.238.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.10.238.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/krb5kdc/rangeradmin_127.10.238.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.10.238.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.10.238.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.10.238.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/krb5kdc/rangerlookup_127.10.238.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.10.238.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/krb5kdc/rangerlookup_127.10.238.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.238.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.238.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.238.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/krb5kdc/HTTP_127.10.238.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.238.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/krb5kdc/HTTP_127.10.238.212@KRBTEST.COM.keytab.
I20260929 10:55:30.704188 11195 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskc60rDW/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/postgres -l logfile start
2026-09-29 10:55:33.325 UTC [21960] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 10:55:33.325 UTC [21960] LOG: listening on IPv4 address "127.10.238.212", port 42831
2026-09-29 10:55:33.328 UTC [21960] LOG: listening on Unix socket "/tmp/.s.PGSQL.42831"
2026-09-29 10:55:33.365 UTC [21963] LOG: database system was shut down at 2026-09-29 10:55:31 UTC
2026-09-29 10:55:33.369 UTC [21960] LOG: database system is ready to accept connections
I20260929 10:55:34.334414 11195 mini_postgres.cc:96] Postgres bound to 42831
2026-09-29 10:55:34.342 UTC [21971] FATAL: database "slave" does not exist
127.10.238.212:42831 - accepting connections
I20260929 10:55:34.342671 11195 mini_ranger.cc:161] Starting Ranger...
I20260929 10:55:34.358199 11195 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 10:55:34.836344 11195 mini_ranger.cc:87] Created ranger Postgres database
I20260929 10:55:34.836457 11195 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ranger-admin
2026-09-29 10:55:35,642 [I] DB FLAVOR :POSTGRES
2026-09-29 10:55:35,643 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 10:55:35,643 [I] Checking connection..
2026-09-29 10:55:35,643 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc60rDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskc60rDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.238.212:42831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 10:55:35,992 [I] Checking connection passed.
2026-09-29 10:55:35,992 [I] --------- Verifying version history table ---------
2026-09-29 10:55:35,992 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc60rDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskc60rDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.238.212:42831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 10:55:36,242 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 10:55:36,243 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc60rDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskc60rDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.238.212:42831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 10:55:36,510 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 10:55:36,510 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 10:55:36,511 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc60rDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskc60rDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.238.212:42831/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskc60rDW/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 10:55:36.774 UTC [22035] WARNING: there is no transaction in progress
2026-09-29 10:55:36,780 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 10:55:36,781 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc60rDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskc60rDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.238.212:42831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 10:55:37,030 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 10:55:37,030 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 10:55:37,030 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc60rDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskc60rDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.238.212:42831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 10:55:37,299 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc60rDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskc60rDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.238.212:42831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 10:55:37,542 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc60rDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskc60rDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.238.212:42831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-8nrt.c.gcp-upstream.internal','N') ;"
2026-09-29 10:55:37,801 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc60rDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskc60rDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.238.212:42831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 10:55:38,066 [I] Table x_portal_user does not exist in database ranger
2026-09-29 10:55:38,067 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc60rDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskc60rDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.238.212:42831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 10:55:38,337 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 10:55:38,338 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc60rDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskc60rDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.238.212:42831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 10:55:38,599 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc60rDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskc60rDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.238.212:42831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 10:55:38,875 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 10:55:38,876 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc60rDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskc60rDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.238.212:42831/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskc60rDW/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 10:55:39.841 UTC [22143] WARNING: there is no transaction in progress
2026-09-29 10:55:39.857 UTC [22143] WARNING: there is no transaction in progress
2026-09-29 10:55:39.873 UTC [22143] WARNING: there is no transaction in progress
2026-09-29 10:55:39.887 UTC [22143] WARNING: there is no transaction in progress
2026-09-29 10:55:39.903 UTC [22143] WARNING: there is no transaction in progress
2026-09-29 10:55:39.920 UTC [22143] WARNING: there is no transaction in progress
2026-09-29 10:55:39.975 UTC [22143] WARNING: there is no transaction in progress
2026-09-29 10:55:39.982 UTC [22143] WARNING: there is no transaction in progress
2026-09-29 10:55:39.996 UTC [22143] WARNING: there is no transaction in progress
2026-09-29 10:55:40.006 UTC [22143] WARNING: there is no transaction in progress
2026-09-29 10:55:40.016 UTC [22143] WARNING: there is no transaction in progress
2026-09-29 10:55:40.029 UTC [22143] WARNING: there is no transaction in progress
2026-09-29 10:55:40.039 UTC [22143] WARNING: there is no transaction in progress
2026-09-29 10:55:40.045 UTC [22143] WARNING: there is no transaction in progress
2026-09-29 10:55:40.052 UTC [22143] WARNING: there is no transaction in progress
2026-09-29 10:55:40.058 UTC [22143] WARNING: there is no transaction in progress
2026-09-29 10:55:40,681 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 10:55:40,681 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc60rDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskc60rDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.238.212:42831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 10:55:40,925 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 10:55:40,926 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc60rDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskc60rDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.238.212:42831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 10:55:41,173 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 10:55:41,173 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc60rDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskc60rDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.238.212:42831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 10:55:41,444 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 10:55:41,445 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc60rDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskc60rDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.238.212:42831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 10:55:41,689 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc60rDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskc60rDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.238.212:42831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 10:55:41,963 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc60rDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskc60rDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.238.212:42831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-8nrt.c.gcp-upstream.internal';"
2026-09-29 10:55:42,213 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 10:55:42,214 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc60rDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskc60rDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.238.212:42831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 10:55:42,482 [I] DB_PATCHES have already been applied
I20260929 10:55:42.488453 11195 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ranger-admin:/tmp/dist-test-taskc60rDW/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskc60rDW/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskc60rDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskc60rDW/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 10:55:42.488517 11195 mini_ranger.cc:193] Using host: 127.10.238.212
I20260929 10:55:42.491711 11195 mini_ranger.cc:227] Ranger admin URL: http://127.10.238.212:54225
Sep 29, 2026 10:55:43 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 10:55:43 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 10:55:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 10:55:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 10:55:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 10:55:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 10:55:43 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 10:55:44.695209 11195 mini_ranger.cc:160] Time spent starting Ranger: real 10.353s user 0.005s sys 0.006s
Sep 29, 2026 10:56:04 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 10:56:04 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 10:56:04 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 10:56:05 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 10:56:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 10:56:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 10:56:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 10:56:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 10:56:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 10:56:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 10:56:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 10:56:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 10:56:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 10:56:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 10:56:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 10:56:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 10:56:05 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 10:56:05 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 10:56:06.570417 11195 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.10.238.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.10.238.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.10.238.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/krb5kdc/rangerkms_127.10.238.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.10.238.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/krb5kdc/rangerkms_127.10.238.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.238.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.10.238.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.238.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/krb5kdc/HTTP_127.10.238.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.238.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/krb5kdc/HTTP_127.10.238.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 10:56:06 dist-test-slave-8nrt krb5kdc[21930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679366, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 10:56:06.665512 11195 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 10:56:06.676313 11195 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 10:56:07.505537 11195 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 10:56:07.505623 11195 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ranger-kms
I20260929 10:56:07.505643 11195 mini_ranger_kms.cc:227] Using postgres at 127.10.238.212:42831
2026-09-29 10:56:08,078 [I] DB FLAVOR :POSTGRES
2026-09-29 10:56:08,078 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 10:56:08,078 [I] Checking connection
2026-09-29 10:56:08,078 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc60rDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.238.212:42831/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 10:56:08,343 [I] connection success
2026-09-29 10:56:08,343 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 10:56:08,343 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 10:56:08,343 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc60rDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.238.212:42831/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 10:56:08,599 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 10:56:08,599 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 10:56:08,599 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 10:56:08,599 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc60rDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.238.212:42831/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 10:56:08,860 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 10:56:09.390441 11195 mini_ranger_kms.cc:310] Created kms service
I20260929 10:56:09.608906 11195 mini_ranger_kms.cc:326] Created kudu user
I20260929 10:56:09.663763 11195 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 10:56:09 dist-test-slave-8nrt krb5kdc[21930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679366, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.10.238.212@KRBTEST.COM
I20260929 10:56:10.089953 11195 mini_ranger_kms.cc:384] Added ranger policy
I20260929 10:56:10.090073 11195 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ranger-kms:/tmp/dist-test-taskc60rDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskc60rDW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskc60rDW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 10:56:10.090095 11195 mini_ranger_kms.cc:240] Using host: 127.10.238.212
I20260929 10:56:10.093873 11195 mini_ranger_kms.cc:276] Ranger KMS PID: 22325
I20260929 10:56:10.094005 11195 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.10.238.212:46511
Sep 29, 2026 10:56:11 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 10:56:11 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 10:56:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 10:56:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 10:56:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 10:56:11 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 29 10:56:12 dist-test-slave-8nrt krb5kdc[21930](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790679372, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.238.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:56:12.402127 11195 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.737s user 0.009s sys 0.212s
I20260929 10:56:12.402237 11195 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 10:56:12.402333 11195 mini_ranger_kms.cc:401] 127.10.238.212:46511/kms/v1/keys
Sep 29 10:56:12 dist-test-slave-8nrt krb5kdc[21930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679372, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.238.212@KRBTEST.COM for HTTP/127.10.238.212@KRBTEST.COM
Sep 29 10:56:13 dist-test-slave-8nrt krb5kdc[21930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679372, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.238.212@KRBTEST.COM for HTTP/127.10.238.212@KRBTEST.COM
Sep 29 10:56:15 dist-test-slave-8nrt krb5kdc[21930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.10.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
I20260929 10:56:15.487670 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:46549
--webserver_interface=127.10.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:37647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:46549
--ranger_config_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.238.212:46511/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:56:15.584097 22375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:56:15.584296 22375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:56:15.584318 22375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:56:15.585479 22375 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:56:15.585520 22375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:56:15.585533 22375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:56:15.585546 22375 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:56:15.585602 22375 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:56:15.587078 22375 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:37647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/master-0/wal
--ranger_kms_url=127.10.238.212:46511/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:46549
--ranger_config_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:46549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:56:15.587332 22375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:56:15.587555 22375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:56:15.592545 22383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:56:15.592602 22375 server_base.cc:1034] running on GCE node
W20260929 10:56:15.592568 22381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:56:15.592571 22380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:56:15.593019 22375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:56:15.593354 22375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:56:15.594516 22375 hybrid_clock.cc:648] HybridClock initialized: now 1790679375594512 us; error 36 us; skew 500 ppm
Sep 29 10:56:15 dist-test-slave-8nrt krb5kdc[21930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679375, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:56:15.596844 22375 init.cc:377] Logged in from keytab as kudu/127.10.238.254@KRBTEST.COM (short username kudu)
I20260929 10:56:15.597244 22375 webserver.cc:460] Webserver started at http://127.10.238.254:36101/ using document root <none> and password file <none>
I20260929 10:56:15.597508 22375 fs_manager.cc:359] Metadata directory not provided
I20260929 10:56:15.597566 22375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:56:15.597708 22375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 10:56:15 dist-test-slave-8nrt krb5kdc[21930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679375, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for HTTP/127.10.238.212@KRBTEST.COM
I20260929 10:56:15.650390 22375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/master-0/data/instance:
uuid: "502db2680c6a4fe5af2c94049e38faad"
format_stamp: "Formatted at 2026-09-29 10:56:15 on dist-test-slave-8nrt"
server_key: "8fa0d506e586c249885404403fb8b018"
server_key_iv: "5d3344a0a55382c0909b270be40dcbad"
server_key_version: "kuduclusterkey@0"
I20260929 10:56:15.650816 22375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/master-0/wal/instance:
uuid: "502db2680c6a4fe5af2c94049e38faad"
format_stamp: "Formatted at 2026-09-29 10:56:15 on dist-test-slave-8nrt"
server_key: "8fa0d506e586c249885404403fb8b018"
server_key_iv: "5d3344a0a55382c0909b270be40dcbad"
server_key_version: "kuduclusterkey@0"
I20260929 10:56:15.652280 22375 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 10:56:15.667027 22390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:56:15.667204 22375 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:56:15.667290 22375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "502db2680c6a4fe5af2c94049e38faad"
format_stamp: "Formatted at 2026-09-29 10:56:15 on dist-test-slave-8nrt"
server_key: "8fa0d506e586c249885404403fb8b018"
server_key_iv: "5d3344a0a55382c0909b270be40dcbad"
server_key_version: "kuduclusterkey@0"
I20260929 10:56:15.667387 22375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:56:15.693406 22375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:56:15.695086 22375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:56:15.695248 22375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:56:15.699555 22375 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.254:46549
I20260929 10:56:15.699606 22442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.254:46549 every 8 connection(s)
I20260929 10:56:15.699952 22375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:56:15.700595 22443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:56:15.702922 22443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 502db2680c6a4fe5af2c94049e38faad: Bootstrap starting.
I20260929 10:56:15.703503 22443 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 502db2680c6a4fe5af2c94049e38faad: Neither blocks nor log segments found. Creating new log.
I20260929 10:56:15.703791 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 22375
I20260929 10:56:15.703820 22443 log.cc:824] T 00000000000000000000000000000000 P 502db2680c6a4fe5af2c94049e38faad: Log is configured to *not* fsync() on all Append() calls
I20260929 10:56:15.703910 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/master-0/wal/instance
I20260929 10:56:15.704178 11195 external_mini_cluster.cc:1428] Setting key a58aff2ccface863a27e2e6a15929a32
I20260929 10:56:15.704576 22443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 502db2680c6a4fe5af2c94049e38faad: No bootstrap required, opened a new log
I20260929 10:56:15.707337 22443 raft_consensus.cc:348] T 00000000000000000000000000000000 P 502db2680c6a4fe5af2c94049e38faad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502db2680c6a4fe5af2c94049e38faad" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 46549 } }
I20260929 10:56:15.707508 22443 raft_consensus.cc:374] T 00000000000000000000000000000000 P 502db2680c6a4fe5af2c94049e38faad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:56:15.707541 22443 raft_consensus.cc:729] T 00000000000000000000000000000000 P 502db2680c6a4fe5af2c94049e38faad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 502db2680c6a4fe5af2c94049e38faad, State: Initialized, Role: FOLLOWER
I20260929 10:56:15.707651 22443 consensus_queue.cc:260] T 00000000000000000000000000000000 P 502db2680c6a4fe5af2c94049e38faad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502db2680c6a4fe5af2c94049e38faad" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 46549 } }
I20260929 10:56:15.707722 22443 raft_consensus.cc:388] T 00000000000000000000000000000000 P 502db2680c6a4fe5af2c94049e38faad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:56:15.707756 22443 raft_consensus.cc:482] T 00000000000000000000000000000000 P 502db2680c6a4fe5af2c94049e38faad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:56:15.707798 22443 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 502db2680c6a4fe5af2c94049e38faad [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:56:15.708459 22443 raft_consensus.cc:504] T 00000000000000000000000000000000 P 502db2680c6a4fe5af2c94049e38faad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502db2680c6a4fe5af2c94049e38faad" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 46549 } }
I20260929 10:56:15.708580 22443 leader_election.cc:302] T 00000000000000000000000000000000 P 502db2680c6a4fe5af2c94049e38faad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 502db2680c6a4fe5af2c94049e38faad; no voters:
I20260929 10:56:15.708791 22443 leader_election.cc:288] T 00000000000000000000000000000000 P 502db2680c6a4fe5af2c94049e38faad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:56:15.708822 22448 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 502db2680c6a4fe5af2c94049e38faad [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:56:15.708933 22448 raft_consensus.cc:686] T 00000000000000000000000000000000 P 502db2680c6a4fe5af2c94049e38faad [term 1 LEADER]: Becoming Leader. State: Replica: 502db2680c6a4fe5af2c94049e38faad, State: Running, Role: LEADER
I20260929 10:56:15.709052 22443 sys_catalog.cc:564] T 00000000000000000000000000000000 P 502db2680c6a4fe5af2c94049e38faad [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:56:15.709084 22448 consensus_queue.cc:237] T 00000000000000000000000000000000 P 502db2680c6a4fe5af2c94049e38faad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502db2680c6a4fe5af2c94049e38faad" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 46549 } }
I20260929 10:56:15.710431 22450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 502db2680c6a4fe5af2c94049e38faad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "502db2680c6a4fe5af2c94049e38faad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502db2680c6a4fe5af2c94049e38faad" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 46549 } } }
I20260929 10:56:15.710516 22450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 502db2680c6a4fe5af2c94049e38faad [sys.catalog]: This master's current role is: LEADER
I20260929 10:56:15.710584 22443 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 10:56:15.710834 22449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 502db2680c6a4fe5af2c94049e38faad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 502db2680c6a4fe5af2c94049e38faad. Latest consensus state: current_term: 1 leader_uuid: "502db2680c6a4fe5af2c94049e38faad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502db2680c6a4fe5af2c94049e38faad" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 46549 } } }
I20260929 10:56:15.710925 22449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 502db2680c6a4fe5af2c94049e38faad [sys.catalog]: This master's current role is: LEADER
I20260929 10:56:15.711076 22462 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 29 10:56:15 dist-test-slave-8nrt krb5kdc[21930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:56:15.715116 22462 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:56:15.718185 22447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:56:15.704741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46356 (local address 127.10.238.254:46549)
0929 10:56:15.704958 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 10:56:15.704967 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 10:56:15.705001 (+ 34us) server_negotiation.cc:407] Connection header received
0929 10:56:15.705907 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:56:15.705916 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:56:15.705980 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:56:15.706125 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:56:15.706760 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:56:15.708204 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:56:15.709275 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:56:15.709967 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:56:15.710092 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:56:15.713318 (+ 3226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:56:15.713386 (+ 68us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:56:15.713386 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:56:15.713493 (+ 107us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:56:15.716434 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:56:15.716883 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:56:15.716886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:56:15.716886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:56:15.716981 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:56:15.717197 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:56:15.717198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:56:15.717198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:56:15.717532 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:56:15.717669 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 10:56:15.717823 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 10:56:15.717976 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":116,"threads_started":1}
I20260929 10:56:15.717722 22462 catalog_manager.cc:1347] Generated new cluster ID: 06b1da774b464c849554c1f6503187cb
I20260929 10:56:15.718820 22462 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:56:15.743510 22462 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:56:15.744259 22462 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:56:15.751030 22462 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 502db2680c6a4fe5af2c94049e38faad: Generated new TSK 0
I20260929 10:56:15.751240 22462 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.238.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab.
Entry for principal kudu/127.10.238.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab.
I20260929 10:56:16.429780 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.238.212:46511/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.193:0
--local_ip_for_outbound_sockets=127.10.238.193
--webserver_interface=127.10.238.193
--webserver_port=0
--tserver_master_addrs=127.10.238.254:46549
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab
--principal=kudu/127.10.238.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:37647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:56:16.529904 22488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:56:16.530149 22488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:56:16.530200 22488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:56:16.531407 22488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:56:16.531478 22488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:56:16.531524 22488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.193
I20260929 10:56:16.532987 22488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:37647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.10.238.212:46511/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab
--principal=kudu/127.10.238.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.238.193
--webserver_port=0
--tserver_master_addrs=127.10.238.254:46549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.193
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:56:16.533268 22488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:56:16.533512 22488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:56:16.538648 22493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:56:16.538744 22496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:56:16.538940 22488 server_base.cc:1034] running on GCE node
W20260929 10:56:16.538981 22494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:56:16.539393 22488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:56:16.539710 22488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:56:16.540887 22488 hybrid_clock.cc:648] HybridClock initialized: now 1790679376540860 us; error 44 us; skew 500 ppm
Sep 29 10:56:16 dist-test-slave-8nrt krb5kdc[21930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679376, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:56:16.543666 22488 init.cc:377] Logged in from keytab as kudu/127.10.238.193@KRBTEST.COM (short username kudu)
I20260929 10:56:16.544186 22488 webserver.cc:460] Webserver started at http://127.10.238.193:36305/ using document root <none> and password file <none>
I20260929 10:56:16.544474 22488 fs_manager.cc:359] Metadata directory not provided
I20260929 10:56:16.544544 22488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:56:16.544683 22488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 10:56:16 dist-test-slave-8nrt krb5kdc[21930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679376, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.193@KRBTEST.COM for HTTP/127.10.238.212@KRBTEST.COM
I20260929 10:56:16.559433 22488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ts-0/data/instance:
uuid: "032ef7ab73944bdd863a7733f0b45627"
format_stamp: "Formatted at 2026-09-29 10:56:16 on dist-test-slave-8nrt"
server_key: "ea8acbe6cbad161e454e3e0d9784705c"
server_key_iv: "bd7a80c1689a81881515af42093475bb"
server_key_version: "kuduclusterkey@0"
I20260929 10:56:16.559892 22488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ts-0/wal/instance:
uuid: "032ef7ab73944bdd863a7733f0b45627"
format_stamp: "Formatted at 2026-09-29 10:56:16 on dist-test-slave-8nrt"
server_key: "ea8acbe6cbad161e454e3e0d9784705c"
server_key_iv: "bd7a80c1689a81881515af42093475bb"
server_key_version: "kuduclusterkey@0"
I20260929 10:56:16.561867 22488 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260929 10:56:16.579944 22503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:56:16.580191 22488 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 10:56:16.580299 22488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ts-0/wal
uuid: "032ef7ab73944bdd863a7733f0b45627"
format_stamp: "Formatted at 2026-09-29 10:56:16 on dist-test-slave-8nrt"
server_key: "ea8acbe6cbad161e454e3e0d9784705c"
server_key_iv: "bd7a80c1689a81881515af42093475bb"
server_key_version: "kuduclusterkey@0"
I20260929 10:56:16.580420 22488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:56:16.606592 22488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:56:16.609107 22488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:56:16.609328 22488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:56:16.609648 22488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:56:16.610059 22488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:56:16.610152 22488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:56:16.610250 22488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:56:16.610328 22488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:56:16.616833 22488 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.193:37123
I20260929 10:56:16.617358 22488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
I20260929 10:56:16.619505 22618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.193:37123 every 8 connection(s)
I20260929 10:56:16.625649 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 22488
I20260929 10:56:16.625761 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ts-0/wal/instance
Sep 29 10:56:16 dist-test-slave-8nrt krb5kdc[21930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679376, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.193@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:56:16.625977 11195 external_mini_cluster.cc:1428] Setting key c0a0e1cce1873c346f641427bdae5a76
I20260929 10:56:16.629969 22622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:56:16.619293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:46549 (local address 127.10.238.193:48351)
0929 10:56:16.619594 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0929 10:56:16.619641 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 10:56:16.620429 (+ 788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:56:16.620768 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:56:16.620773 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:56:16.621020 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:56:16.621353 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:56:16.621360 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:56:16.622365 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:56:16.622366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:56:16.623517 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:56:16.623519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:56:16.623924 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:56:16.623925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:56:16.624044 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:56:16.624619 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:56:16.624627 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:56:16.626164 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:56:16.628286 (+ 2122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:56:16.628293 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:56:16.628295 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:56:16.628585 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:56:16.628824 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:56:16.628825 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:56:16.628825 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:56:16.628954 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:56:16.629250 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:56:16.629293 (+ 43us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:56:16.629454 (+ 161us) client_negotiation.cc:769] Sending connection context
0929 10:56:16.629569 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 10:56:16.629721 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":93,"threads_started":1}
I20260929 10:56:16.630560 22621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:56:16.619291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:48351 (local address 127.10.238.254:46549)
0929 10:56:16.619809 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0929 10:56:16.619812 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:56:16.619850 (+ 38us) server_negotiation.cc:407] Connection header received
0929 10:56:16.620563 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:56:16.620564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:56:16.620579 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:56:16.620674 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:56:16.621478 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:56:16.622236 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:56:16.623613 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:56:16.623822 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:56:16.623911 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:56:16.626309 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:56:16.626310 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:56:16.626311 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:56:16.626318 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:56:16.628186 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:56:16.628674 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:56:16.628676 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:56:16.628676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:56:16.628742 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:56:16.629029 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:56:16.629030 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:56:16.629031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:56:16.629166 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:56:16.629225 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 10:56:16.630361 (+ 1136us) server_negotiation.cc:299] Negotiation successful
0929 10:56:16.630447 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":73,"threads_started":1}
I20260929 10:56:16.630981 22619 heartbeater.cc:347] Connected to a master server at 127.10.238.254:46549
I20260929 10:56:16.631071 22619 heartbeater.cc:464] Registering TS with master...
I20260929 10:56:16.631245 22619 heartbeater.cc:511] Master 127.10.238.254:46549 requested a full tablet report, sending...
I20260929 10:56:16.631770 22407 ts_manager.cc:194] Registered new tserver with Master: 032ef7ab73944bdd863a7733f0b45627 (127.10.238.193:37123)
I20260929 10:56:16.632671 22407 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.193@KRBTEST.COM'} at 127.10.238.193:48351
WARNING: no policy specified for kudu/127.10.238.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab.
Entry for principal kudu/127.10.238.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab.
I20260929 10:56:16.661378 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.238.212:46511/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.194:0
--local_ip_for_outbound_sockets=127.10.238.194
--webserver_interface=127.10.238.194
--webserver_port=0
--tserver_master_addrs=127.10.238.254:46549
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab
--principal=kudu/127.10.238.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:37647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 10:56:16 dist-test-slave-8nrt krb5kdc[21930](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790679376, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 10:56:16.782148 22625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:56:16.782502 22625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:56:16.782582 22625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:56:16.784323 22625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:56:16.784430 22625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:56:16.784511 22625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.194
I20260929 10:56:16.786710 22625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:37647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.10.238.212:46511/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab
--principal=kudu/127.10.238.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.238.194
--webserver_port=0
--tserver_master_addrs=127.10.238.254:46549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.194
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:56:16.787189 22625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:56:16.787505 22625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:56:16.795265 22635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:56:16.797303 22625 server_base.cc:1034] running on GCE node
W20260929 10:56:16.798961 22633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:56:16.803424 22632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:56:16.803779 22625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:56:16.804075 22625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:56:16.805230 22625 hybrid_clock.cc:648] HybridClock initialized: now 1790679376805204 us; error 43 us; skew 500 ppm
Sep 29 10:56:16 dist-test-slave-8nrt krb5kdc[21930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679376, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:56:16.808458 22625 init.cc:377] Logged in from keytab as kudu/127.10.238.194@KRBTEST.COM (short username kudu)
I20260929 10:56:16.808987 22625 webserver.cc:460] Webserver started at http://127.10.238.194:45259/ using document root <none> and password file <none>
I20260929 10:56:16.809333 22625 fs_manager.cc:359] Metadata directory not provided
I20260929 10:56:16.809486 22625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:56:16.809666 22625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 10:56:16 dist-test-slave-8nrt krb5kdc[21930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679376, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.194@KRBTEST.COM for HTTP/127.10.238.212@KRBTEST.COM
I20260929 10:56:16.823045 22625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ts-1/data/instance:
uuid: "f60a102acfbd4f6d9057d66c7d30f3fa"
format_stamp: "Formatted at 2026-09-29 10:56:16 on dist-test-slave-8nrt"
server_key: "beff05324d29cf7272f07a75483b018e"
server_key_iv: "726a3b54ad0495667cdf7d753476a91b"
server_key_version: "kuduclusterkey@0"
I20260929 10:56:16.823582 22625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ts-1/wal/instance:
uuid: "f60a102acfbd4f6d9057d66c7d30f3fa"
format_stamp: "Formatted at 2026-09-29 10:56:16 on dist-test-slave-8nrt"
server_key: "beff05324d29cf7272f07a75483b018e"
server_key_iv: "726a3b54ad0495667cdf7d753476a91b"
server_key_version: "kuduclusterkey@0"
I20260929 10:56:16.825431 22625 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 10:56:16.841394 22643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:56:16.841626 22625 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 10:56:16.841743 22625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ts-1/wal
uuid: "f60a102acfbd4f6d9057d66c7d30f3fa"
format_stamp: "Formatted at 2026-09-29 10:56:16 on dist-test-slave-8nrt"
server_key: "beff05324d29cf7272f07a75483b018e"
server_key_iv: "726a3b54ad0495667cdf7d753476a91b"
server_key_version: "kuduclusterkey@0"
I20260929 10:56:16.841900 22625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:56:16.876135 22625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:56:16.878697 22625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:56:16.878917 22625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:56:16.879192 22625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:56:16.879603 22625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:56:16.879686 22625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:56:16.879750 22625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:56:16.879797 22625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:56:16.887562 22625 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.194:44521
I20260929 10:56:16.887686 22756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.194:44521 every 8 connection(s)
I20260929 10:56:16.888059 22625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
I20260929 10:56:16.890820 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 22625
I20260929 10:56:16.890945 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ts-1/wal/instance
I20260929 10:56:16.891145 11195 external_mini_cluster.cc:1428] Setting key 94d52f186703e55858da505f62112ba4
Sep 29 10:56:16 dist-test-slave-8nrt krb5kdc[21930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679376, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.194@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:56:16.909766 22621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:56:16.888880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:56717 (local address 127.10.238.254:46549)
0929 10:56:16.888931 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 10:56:16.888933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:56:16.897175 (+ 8242us) server_negotiation.cc:407] Connection header received
0929 10:56:16.899678 (+ 2503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:56:16.899680 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:56:16.899701 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:56:16.899735 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:56:16.900666 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:56:16.901466 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:56:16.902970 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:56:16.903296 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:56:16.903339 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:56:16.906182 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:56:16.906184 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:56:16.906184 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:56:16.906192 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:56:16.908008 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:56:16.908514 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:56:16.908515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:56:16.908515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:56:16.908576 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:56:16.908894 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:56:16.908895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:56:16.908896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:56:16.909054 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:56:16.909098 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0929 10:56:16.909551 (+ 453us) server_negotiation.cc:299] Negotiation successful
0929 10:56:16.909668 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 10:56:16.910641 22759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:56:16.888867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:46549 (local address 127.10.238.194:56717)
0929 10:56:16.897030 (+ 8163us) negotiation.cc:107] Waiting for socket to connect
0929 10:56:16.897087 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 10:56:16.899530 (+ 2443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:56:16.899967 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:56:16.899972 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:56:16.900239 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:56:16.900544 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:56:16.900553 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:56:16.901586 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:56:16.901587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:56:16.902866 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:56:16.902869 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:56:16.903431 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:56:16.903432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:56:16.903574 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:56:16.904228 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:56:16.904240 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:56:16.906061 (+ 1821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:56:16.908103 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:56:16.908110 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:56:16.908112 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:56:16.908418 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:56:16.908654 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:56:16.908654 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:56:16.908655 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:56:16.908795 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:56:16.909161 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:56:16.909168 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:56:16.909386 (+ 218us) client_negotiation.cc:769] Sending connection context
0929 10:56:16.910155 (+ 769us) client_negotiation.cc:240] Negotiation successful
0929 10:56:16.910395 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8104,"thread_start_us":95,"threads_started":1}
I20260929 10:56:16.911355 22757 heartbeater.cc:347] Connected to a master server at 127.10.238.254:46549
I20260929 10:56:16.911470 22757 heartbeater.cc:464] Registering TS with master...
I20260929 10:56:16.911692 22757 heartbeater.cc:511] Master 127.10.238.254:46549 requested a full tablet report, sending...
I20260929 10:56:16.912088 22407 ts_manager.cc:194] Registered new tserver with Master: f60a102acfbd4f6d9057d66c7d30f3fa (127.10.238.194:44521)
I20260929 10:56:16.912569 22407 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.194@KRBTEST.COM'} at 127.10.238.194:56717
WARNING: no policy specified for kudu/127.10.238.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab.
Entry for principal kudu/127.10.238.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab.
I20260929 10:56:16.942333 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.238.212:46511/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.195:0
--local_ip_for_outbound_sockets=127.10.238.195
--webserver_interface=127.10.238.195
--webserver_port=0
--tserver_master_addrs=127.10.238.254:46549
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab
--principal=kudu/127.10.238.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:37647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:56:17.092120 22762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:56:17.092460 22762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:56:17.092551 22762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:56:17.094297 22762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:56:17.094507 22762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:56:17.094594 22762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.195
I20260929 10:56:17.096771 22762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:37647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.10.238.212:46511/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab
--principal=kudu/127.10.238.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.238.195
--webserver_port=0
--tserver_master_addrs=127.10.238.254:46549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.195
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:56:17.097221 22762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:56:17.097556 22762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:56:17.104142 22768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:56:17.104552 22767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:56:17.104840 22770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:56:17.106107 22762 server_base.cc:1034] running on GCE node
I20260929 10:56:17.106523 22762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:56:17.106830 22762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:56:17.108026 22762 hybrid_clock.cc:648] HybridClock initialized: now 1790679377108001 us; error 41 us; skew 500 ppm
Sep 29 10:56:17 dist-test-slave-8nrt krb5kdc[21930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679377, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:56:17.117125 22762 init.cc:377] Logged in from keytab as kudu/127.10.238.195@KRBTEST.COM (short username kudu)
I20260929 10:56:17.117743 22762 webserver.cc:460] Webserver started at http://127.10.238.195:36045/ using document root <none> and password file <none>
I20260929 10:56:17.118036 22762 fs_manager.cc:359] Metadata directory not provided
I20260929 10:56:17.118103 22762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:56:17.118237 22762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 10:56:17 dist-test-slave-8nrt krb5kdc[21930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679377, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.195@KRBTEST.COM for HTTP/127.10.238.212@KRBTEST.COM
I20260929 10:56:17.139899 22762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ts-2/data/instance:
uuid: "84c4fa7db2ba482d89a6d60cb5b9a207"
format_stamp: "Formatted at 2026-09-29 10:56:17 on dist-test-slave-8nrt"
server_key: "2fd3ece2403c847b6d8fcfe3234de3bd"
server_key_iv: "47ed7039178cf15c395da666da6419f8"
server_key_version: "kuduclusterkey@0"
I20260929 10:56:17.140404 22762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ts-2/wal/instance:
uuid: "84c4fa7db2ba482d89a6d60cb5b9a207"
format_stamp: "Formatted at 2026-09-29 10:56:17 on dist-test-slave-8nrt"
server_key: "2fd3ece2403c847b6d8fcfe3234de3bd"
server_key_iv: "47ed7039178cf15c395da666da6419f8"
server_key_version: "kuduclusterkey@0"
I20260929 10:56:17.149694 22762 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.000s sys 0.004s
I20260929 10:56:17.166181 22777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:56:17.166419 22762 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 10:56:17.166531 22762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ts-2/wal
uuid: "84c4fa7db2ba482d89a6d60cb5b9a207"
format_stamp: "Formatted at 2026-09-29 10:56:17 on dist-test-slave-8nrt"
server_key: "2fd3ece2403c847b6d8fcfe3234de3bd"
server_key_iv: "47ed7039178cf15c395da666da6419f8"
server_key_version: "kuduclusterkey@0"
I20260929 10:56:17.166649 22762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:56:17.224082 22762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:56:17.228209 22762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:56:17.228406 22762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:56:17.228639 22762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:56:17.229038 22762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:56:17.229090 22762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:56:17.229126 22762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:56:17.229149 22762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:56:17.236441 22762 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.195:46121
I20260929 10:56:17.236912 22762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
I20260929 10:56:17.239843 22890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.195:46121 every 8 connection(s)
I20260929 10:56:17.241734 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 22762
I20260929 10:56:17.241844 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ts-2/wal/instance
I20260929 10:56:17.242060 11195 external_mini_cluster.cc:1428] Setting key 05f9c6c86a16ae5147a5e5c90967c997
Sep 29 10:56:17 dist-test-slave-8nrt krb5kdc[21930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679377, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.195@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:56:17.253435 22621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:56:17.240789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:44351 (local address 127.10.238.254:46549)
0929 10:56:17.240839 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 10:56:17.240842 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:56:17.241447 (+ 605us) server_negotiation.cc:407] Connection header received
0929 10:56:17.242282 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:56:17.242283 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:56:17.242297 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:56:17.242331 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:56:17.243198 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:56:17.244083 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:56:17.246581 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:56:17.246922 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:56:17.246970 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:56:17.250000 (+ 3030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:56:17.250001 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:56:17.250002 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:56:17.250010 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:56:17.251753 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:56:17.252281 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:56:17.252282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:56:17.252282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:56:17.252341 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:56:17.252611 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:56:17.252612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:56:17.252612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:56:17.252761 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:56:17.252817 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 10:56:17.253194 (+ 377us) server_negotiation.cc:299] Negotiation successful
0929 10:56:17.253335 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 10:56:17.254006 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:56:17.240482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:46549 (local address 127.10.238.195:44351)
0929 10:56:17.241212 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0929 10:56:17.241347 (+ 135us) client_negotiation.cc:174] Beginning negotiation
0929 10:56:17.242128 (+ 781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:56:17.242536 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:56:17.242541 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:56:17.242796 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:56:17.243079 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:56:17.243085 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:56:17.244230 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:56:17.244231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:56:17.246470 (+ 2239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:56:17.246473 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:56:17.247064 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:56:17.247065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:56:17.247213 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:56:17.247937 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:56:17.247948 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:56:17.249882 (+ 1934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:56:17.251855 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:56:17.251861 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:56:17.251864 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:56:17.252179 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:56:17.252408 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:56:17.252409 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:56:17.252409 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:56:17.252539 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:56:17.252853 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:56:17.252859 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:56:17.253025 (+ 166us) client_negotiation.cc:769] Sending connection context
0929 10:56:17.253584 (+ 559us) client_negotiation.cc:240] Negotiation successful
0929 10:56:17.253758 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":88,"threads_started":1}
I20260929 10:56:17.254441 22891 heartbeater.cc:347] Connected to a master server at 127.10.238.254:46549
I20260929 10:56:17.254541 22891 heartbeater.cc:464] Registering TS with master...
I20260929 10:56:17.254742 22891 heartbeater.cc:511] Master 127.10.238.254:46549 requested a full tablet report, sending...
I20260929 10:56:17.255088 22407 ts_manager.cc:194] Registered new tserver with Master: 84c4fa7db2ba482d89a6d60cb5b9a207 (127.10.238.195:46121)
I20260929 10:56:17.255635 22407 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.195@KRBTEST.COM'} at 127.10.238.195:44351
I20260929 10:56:17.257380 11195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 10:56:17 dist-test-slave-8nrt krb5kdc[21930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679376, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for HTTP/127.10.238.212@KRBTEST.COM
I20260929 10:56:17.288298 22621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:56:17.259241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46366 (local address 127.10.238.254:46549)
0929 10:56:17.259743 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0929 10:56:17.259745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:56:17.260020 (+ 275us) server_negotiation.cc:407] Connection header received
0929 10:56:17.260112 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:56:17.260113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:56:17.260123 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:56:17.260150 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:56:17.260895 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:56:17.261618 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:56:17.273769 (+ 12151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:56:17.274125 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:56:17.274214 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:56:17.278341 (+ 4127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:56:17.278344 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:56:17.278344 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:56:17.278353 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:56:17.280203 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:56:17.284847 (+ 4644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:56:17.284848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:56:17.284849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:56:17.284914 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:56:17.287633 (+ 2719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:56:17.287635 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:56:17.287635 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:56:17.287792 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:56:17.287882 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 10:56:17.288093 (+ 211us) server_negotiation.cc:299] Negotiation successful
0929 10:56:17.288183 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478}
I20260929 10:56:17.290431 22407 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46366:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:56:17.293593 22407 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:56:17.304023 22906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:56:17.299130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57564 (local address 127.10.238.193:37123)
0929 10:56:17.299283 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 10:56:17.299290 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:56:17.299745 (+ 455us) server_negotiation.cc:407] Connection header received
0929 10:56:17.299798 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:56:17.299800 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:56:17.299819 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:56:17.299898 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:56:17.300110 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:56:17.301228 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:56:17.302804 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:56:17.303537 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:56:17.303588 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:56:17.303648 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 10:56:17.303912 (+ 264us) server_negotiation.cc:299] Negotiation successful
0929 10:56:17.303950 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":54,"threads_started":1}
I20260929 10:56:17.304008 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:56:17.298934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.195:46121 (local address 127.0.0.1:43322)
0929 10:56:17.299352 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0929 10:56:17.299575 (+ 223us) client_negotiation.cc:174] Beginning negotiation
0929 10:56:17.299646 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:56:17.300337 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:56:17.300338 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:56:17.300339 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:56:17.300378 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:56:17.300379 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:56:17.301610 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:56:17.301611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:56:17.302698 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:56:17.302701 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:56:17.303700 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:56:17.303702 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:56:17.303787 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:56:17.303800 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 10:56:17.303867 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 10:56:17.303924 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"mutex_wait_us":5,"thread_start_us":87,"threads_started":1}
I20260929 10:56:17.304267 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:56:17.299147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43322 (local address 127.10.238.195:46121)
0929 10:56:17.299614 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0929 10:56:17.299618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:56:17.299631 (+ 13us) server_negotiation.cc:407] Connection header received
0929 10:56:17.300158 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:56:17.300161 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:56:17.300180 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:56:17.300262 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:56:17.300451 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:56:17.301512 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:56:17.302823 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:56:17.303534 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:56:17.303620 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:56:17.303666 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 10:56:17.304133 (+ 467us) server_negotiation.cc:299] Negotiation successful
0929 10:56:17.304182 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":95,"threads_started":1}
I20260929 10:56:17.304716 22904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:56:17.299023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:37123 (local address 127.0.0.1:57564)
0929 10:56:17.299680 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0929 10:56:17.299689 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 10:56:17.299743 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:56:17.299950 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:56:17.299953 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:56:17.299955 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:56:17.300022 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:56:17.300023 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:56:17.301342 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:56:17.301343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:56:17.302704 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:56:17.302706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:56:17.303718 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:56:17.303718 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:56:17.303787 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:56:17.303800 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 10:56:17.304576 (+ 776us) client_negotiation.cc:240] Negotiation successful
0929 10:56:17.304628 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"mutex_wait_us":3,"thread_start_us":81,"threads_started":1}
I20260929 10:56:17.305686 22542 tablet_service.cc:1467] Processing CreateTablet for tablet 6dbd380524db4aab970de4809686f7eb (DEFAULT_TABLE table=test-table [id=1cbaf9c2b3b94ecab68984a56287431e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:56:17.306133 22542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dbd380524db4aab970de4809686f7eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:56:17.306504 22825 tablet_service.cc:1467] Processing CreateTablet for tablet 6dbd380524db4aab970de4809686f7eb (DEFAULT_TABLE table=test-table [id=1cbaf9c2b3b94ecab68984a56287431e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:56:17.306841 22825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dbd380524db4aab970de4809686f7eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:56:17.315665 22905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:56:17.299224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.194:44521 (local address 127.0.0.1:53488)
0929 10:56:17.299828 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0929 10:56:17.299837 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 10:56:17.299876 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:56:17.311744 (+ 11868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:56:17.311745 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:56:17.311748 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:56:17.311804 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:56:17.311806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:56:17.313162 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:56:17.313163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:56:17.314404 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:56:17.314407 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:56:17.315432 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:56:17.315433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:56:17.315500 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:56:17.315506 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 10:56:17.315536 (+ 30us) client_negotiation.cc:240] Negotiation successful
0929 10:56:17.315580 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":63,"threads_started":1}
I20260929 10:56:17.316035 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:56:17.299417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53488 (local address 127.10.238.194:44521)
0929 10:56:17.311526 (+ 12109us) server_negotiation.cc:206] Beginning negotiation
0929 10:56:17.311532 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:56:17.311552 (+ 20us) server_negotiation.cc:407] Connection header received
0929 10:56:17.311599 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:56:17.311601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:56:17.311618 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:56:17.311877 (+ 259us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:56:17.311952 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:56:17.313056 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:56:17.314536 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:56:17.315324 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:56:17.315823 (+ 499us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:56:17.315894 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 10:56:17.315920 (+ 26us) server_negotiation.cc:299] Negotiation successful
0929 10:56:17.315958 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12063,"thread_start_us":82,"threads_started":1}
I20260929 10:56:17.316740 22677 tablet_service.cc:1467] Processing CreateTablet for tablet 6dbd380524db4aab970de4809686f7eb (DEFAULT_TABLE table=test-table [id=1cbaf9c2b3b94ecab68984a56287431e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:56:17.317001 22911 tablet_bootstrap.cc:492] T 6dbd380524db4aab970de4809686f7eb P 84c4fa7db2ba482d89a6d60cb5b9a207: Bootstrap starting.
I20260929 10:56:17.317101 22677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dbd380524db4aab970de4809686f7eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:56:17.318073 22911 tablet_bootstrap.cc:654] T 6dbd380524db4aab970de4809686f7eb P 84c4fa7db2ba482d89a6d60cb5b9a207: Neither blocks nor log segments found. Creating new log.
I20260929 10:56:17.318403 22911 log.cc:824] T 6dbd380524db4aab970de4809686f7eb P 84c4fa7db2ba482d89a6d60cb5b9a207: Log is configured to *not* fsync() on all Append() calls
I20260929 10:56:17.318796 22910 tablet_bootstrap.cc:492] T 6dbd380524db4aab970de4809686f7eb P 032ef7ab73944bdd863a7733f0b45627: Bootstrap starting.
I20260929 10:56:17.319602 22910 tablet_bootstrap.cc:654] T 6dbd380524db4aab970de4809686f7eb P 032ef7ab73944bdd863a7733f0b45627: Neither blocks nor log segments found. Creating new log.
I20260929 10:56:17.319897 22910 log.cc:824] T 6dbd380524db4aab970de4809686f7eb P 032ef7ab73944bdd863a7733f0b45627: Log is configured to *not* fsync() on all Append() calls
I20260929 10:56:17.322201 22911 tablet_bootstrap.cc:492] T 6dbd380524db4aab970de4809686f7eb P 84c4fa7db2ba482d89a6d60cb5b9a207: No bootstrap required, opened a new log
I20260929 10:56:17.322293 22911 ts_tablet_manager.cc:1397] T 6dbd380524db4aab970de4809686f7eb P 84c4fa7db2ba482d89a6d60cb5b9a207: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260929 10:56:17.324849 22911 raft_consensus.cc:348] T 6dbd380524db4aab970de4809686f7eb P 84c4fa7db2ba482d89a6d60cb5b9a207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c4fa7db2ba482d89a6d60cb5b9a207" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 46121 } } peers { permanent_uuid: "f60a102acfbd4f6d9057d66c7d30f3fa" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 44521 } } peers { permanent_uuid: "032ef7ab73944bdd863a7733f0b45627" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 37123 } }
I20260929 10:56:17.325042 22911 raft_consensus.cc:374] T 6dbd380524db4aab970de4809686f7eb P 84c4fa7db2ba482d89a6d60cb5b9a207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:56:17.325076 22911 raft_consensus.cc:729] T 6dbd380524db4aab970de4809686f7eb P 84c4fa7db2ba482d89a6d60cb5b9a207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c4fa7db2ba482d89a6d60cb5b9a207, State: Initialized, Role: FOLLOWER
I20260929 10:56:17.325243 22911 consensus_queue.cc:260] T 6dbd380524db4aab970de4809686f7eb P 84c4fa7db2ba482d89a6d60cb5b9a207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c4fa7db2ba482d89a6d60cb5b9a207" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 46121 } } peers { permanent_uuid: "f60a102acfbd4f6d9057d66c7d30f3fa" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 44521 } } peers { permanent_uuid: "032ef7ab73944bdd863a7733f0b45627" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 37123 } }
I20260929 10:56:17.325543 22911 ts_tablet_manager.cc:1428] T 6dbd380524db4aab970de4809686f7eb P 84c4fa7db2ba482d89a6d60cb5b9a207: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260929 10:56:17.325664 22891 heartbeater.cc:503] Master 127.10.238.254:46549 was elected leader, sending a full tablet report...
I20260929 10:56:17.326967 22914 tablet_bootstrap.cc:492] T 6dbd380524db4aab970de4809686f7eb P f60a102acfbd4f6d9057d66c7d30f3fa: Bootstrap starting.
I20260929 10:56:17.328094 22914 tablet_bootstrap.cc:654] T 6dbd380524db4aab970de4809686f7eb P f60a102acfbd4f6d9057d66c7d30f3fa: Neither blocks nor log segments found. Creating new log.
I20260929 10:56:17.328436 22914 log.cc:824] T 6dbd380524db4aab970de4809686f7eb P f60a102acfbd4f6d9057d66c7d30f3fa: Log is configured to *not* fsync() on all Append() calls
I20260929 10:56:17.329084 22910 tablet_bootstrap.cc:492] T 6dbd380524db4aab970de4809686f7eb P 032ef7ab73944bdd863a7733f0b45627: No bootstrap required, opened a new log
I20260929 10:56:17.329172 22910 ts_tablet_manager.cc:1397] T 6dbd380524db4aab970de4809686f7eb P 032ef7ab73944bdd863a7733f0b45627: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20260929 10:56:17.332228 22910 raft_consensus.cc:348] T 6dbd380524db4aab970de4809686f7eb P 032ef7ab73944bdd863a7733f0b45627 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c4fa7db2ba482d89a6d60cb5b9a207" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 46121 } } peers { permanent_uuid: "f60a102acfbd4f6d9057d66c7d30f3fa" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 44521 } } peers { permanent_uuid: "032ef7ab73944bdd863a7733f0b45627" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 37123 } }
I20260929 10:56:17.332413 22910 raft_consensus.cc:374] T 6dbd380524db4aab970de4809686f7eb P 032ef7ab73944bdd863a7733f0b45627 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:56:17.332444 22910 raft_consensus.cc:729] T 6dbd380524db4aab970de4809686f7eb P 032ef7ab73944bdd863a7733f0b45627 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 032ef7ab73944bdd863a7733f0b45627, State: Initialized, Role: FOLLOWER
I20260929 10:56:17.332600 22910 consensus_queue.cc:260] T 6dbd380524db4aab970de4809686f7eb P 032ef7ab73944bdd863a7733f0b45627 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c4fa7db2ba482d89a6d60cb5b9a207" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 46121 } } peers { permanent_uuid: "f60a102acfbd4f6d9057d66c7d30f3fa" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 44521 } } peers { permanent_uuid: "032ef7ab73944bdd863a7733f0b45627" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 37123 } }
I20260929 10:56:17.332870 22910 ts_tablet_manager.cc:1428] T 6dbd380524db4aab970de4809686f7eb P 032ef7ab73944bdd863a7733f0b45627: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 10:56:17.332998 22619 heartbeater.cc:503] Master 127.10.238.254:46549 was elected leader, sending a full tablet report...
I20260929 10:56:17.335173 22914 tablet_bootstrap.cc:492] T 6dbd380524db4aab970de4809686f7eb P f60a102acfbd4f6d9057d66c7d30f3fa: No bootstrap required, opened a new log
I20260929 10:56:17.335264 22914 ts_tablet_manager.cc:1397] T 6dbd380524db4aab970de4809686f7eb P f60a102acfbd4f6d9057d66c7d30f3fa: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260929 10:56:17.337752 22914 raft_consensus.cc:348] T 6dbd380524db4aab970de4809686f7eb P f60a102acfbd4f6d9057d66c7d30f3fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c4fa7db2ba482d89a6d60cb5b9a207" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 46121 } } peers { permanent_uuid: "f60a102acfbd4f6d9057d66c7d30f3fa" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 44521 } } peers { permanent_uuid: "032ef7ab73944bdd863a7733f0b45627" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 37123 } }
I20260929 10:56:17.337930 22914 raft_consensus.cc:374] T 6dbd380524db4aab970de4809686f7eb P f60a102acfbd4f6d9057d66c7d30f3fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:56:17.337961 22914 raft_consensus.cc:729] T 6dbd380524db4aab970de4809686f7eb P f60a102acfbd4f6d9057d66c7d30f3fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f60a102acfbd4f6d9057d66c7d30f3fa, State: Initialized, Role: FOLLOWER
I20260929 10:56:17.338100 22914 consensus_queue.cc:260] T 6dbd380524db4aab970de4809686f7eb P f60a102acfbd4f6d9057d66c7d30f3fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c4fa7db2ba482d89a6d60cb5b9a207" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 46121 } } peers { permanent_uuid: "f60a102acfbd4f6d9057d66c7d30f3fa" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 44521 } } peers { permanent_uuid: "032ef7ab73944bdd863a7733f0b45627" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 37123 } }
I20260929 10:56:17.338364 22914 ts_tablet_manager.cc:1428] T 6dbd380524db4aab970de4809686f7eb P f60a102acfbd4f6d9057d66c7d30f3fa: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260929 10:56:17.338488 22757 heartbeater.cc:503] Master 127.10.238.254:46549 was elected leader, sending a full tablet report...
W20260929 10:56:17.370158 22620 tablet.cc:2369] T 6dbd380524db4aab970de4809686f7eb P 032ef7ab73944bdd863a7733f0b45627: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 10:56:17.389024 22758 tablet.cc:2369] T 6dbd380524db4aab970de4809686f7eb P f60a102acfbd4f6d9057d66c7d30f3fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 10:56:17.491248 22892 tablet.cc:2369] T 6dbd380524db4aab970de4809686f7eb P 84c4fa7db2ba482d89a6d60cb5b9a207: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 10:56:17 dist-test-slave-8nrt krb5kdc[21930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679376, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for HTTP/127.10.238.212@KRBTEST.COM
I20260929 10:56:17.549310 22917 raft_consensus.cc:482] T 6dbd380524db4aab970de4809686f7eb P 032ef7ab73944bdd863a7733f0b45627 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:56:17.549470 22917 raft_consensus.cc:504] T 6dbd380524db4aab970de4809686f7eb P 032ef7ab73944bdd863a7733f0b45627 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c4fa7db2ba482d89a6d60cb5b9a207" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 46121 } } peers { permanent_uuid: "f60a102acfbd4f6d9057d66c7d30f3fa" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 44521 } } peers { permanent_uuid: "032ef7ab73944bdd863a7733f0b45627" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 37123 } }
I20260929 10:56:17.549855 22917 leader_election.cc:288] T 6dbd380524db4aab970de4809686f7eb P 032ef7ab73944bdd863a7733f0b45627 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84c4fa7db2ba482d89a6d60cb5b9a207 (127.10.238.195:46121), f60a102acfbd4f6d9057d66c7d30f3fa (127.10.238.194:44521)
I20260929 10:56:17.555167 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:56:17.550137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:60393 (local address 127.10.238.195:46121)
0929 10:56:17.550429 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0929 10:56:17.550431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:56:17.550800 (+ 369us) server_negotiation.cc:407] Connection header received
0929 10:56:17.550909 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:56:17.550910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:56:17.550923 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:56:17.550953 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:56:17.551192 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:56:17.551877 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:56:17.554263 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:56:17.554783 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:56:17.554834 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:56:17.554843 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 10:56:17.555057 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 10:56:17.555097 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263}
I20260929 10:56:17.555578 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:56:17.550072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.195:46121 (local address 127.10.238.193:60393)
0929 10:56:17.550736 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0929 10:56:17.550747 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 10:56:17.550830 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:56:17.551021 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:56:17.551022 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:56:17.551035 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:56:17.551119 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:56:17.551121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:56:17.552000 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:56:17.552001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:56:17.554157 (+ 2156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:56:17.554160 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:56:17.554907 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:56:17.554907 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:56:17.554981 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:56:17.554985 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 10:56:17.555446 (+ 461us) client_negotiation.cc:240] Negotiation successful
0929 10:56:17.555493 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":107,"threads_started":1}
I20260929 10:56:17.557657 22915 raft_consensus.cc:482] T 6dbd380524db4aab970de4809686f7eb P 84c4fa7db2ba482d89a6d60cb5b9a207 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:56:17.557703 22844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbd380524db4aab970de4809686f7eb" candidate_uuid: "032ef7ab73944bdd863a7733f0b45627" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c4fa7db2ba482d89a6d60cb5b9a207" is_pre_election: true
I20260929 10:56:17.557763 22915 raft_consensus.cc:504] T 6dbd380524db4aab970de4809686f7eb P 84c4fa7db2ba482d89a6d60cb5b9a207 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c4fa7db2ba482d89a6d60cb5b9a207" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 46121 } } peers { permanent_uuid: "f60a102acfbd4f6d9057d66c7d30f3fa" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 44521 } } peers { permanent_uuid: "032ef7ab73944bdd863a7733f0b45627" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 37123 } }
I20260929 10:56:17.557883 22844 raft_consensus.cc:2454] T 6dbd380524db4aab970de4809686f7eb P 84c4fa7db2ba482d89a6d60cb5b9a207 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 032ef7ab73944bdd863a7733f0b45627 in term 0.
I20260929 10:56:17.558103 22915 leader_election.cc:288] T 6dbd380524db4aab970de4809686f7eb P 84c4fa7db2ba482d89a6d60cb5b9a207 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f60a102acfbd4f6d9057d66c7d30f3fa (127.10.238.194:44521), 032ef7ab73944bdd863a7733f0b45627 (127.10.238.193:37123)
I20260929 10:56:17.563954 22505 leader_election.cc:302] T 6dbd380524db4aab970de4809686f7eb P 032ef7ab73944bdd863a7733f0b45627 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 032ef7ab73944bdd863a7733f0b45627, 84c4fa7db2ba482d89a6d60cb5b9a207; no voters:
I20260929 10:56:17.564131 22917 raft_consensus.cc:2781] T 6dbd380524db4aab970de4809686f7eb P 032ef7ab73944bdd863a7733f0b45627 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:56:17.564186 22917 raft_consensus.cc:482] T 6dbd380524db4aab970de4809686f7eb P 032ef7ab73944bdd863a7733f0b45627 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:56:17.564209 22917 raft_consensus.cc:3037] T 6dbd380524db4aab970de4809686f7eb P 032ef7ab73944bdd863a7733f0b45627 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:56:17.565233 22917 raft_consensus.cc:504] T 6dbd380524db4aab970de4809686f7eb P 032ef7ab73944bdd863a7733f0b45627 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c4fa7db2ba482d89a6d60cb5b9a207" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 46121 } } peers { permanent_uuid: "f60a102acfbd4f6d9057d66c7d30f3fa" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 44521 } } peers { permanent_uuid: "032ef7ab73944bdd863a7733f0b45627" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 37123 } }
I20260929 10:56:17.565414 22917 leader_election.cc:288] T 6dbd380524db4aab970de4809686f7eb P 032ef7ab73944bdd863a7733f0b45627 [CANDIDATE]: Term 1 election: Requested vote from peers 84c4fa7db2ba482d89a6d60cb5b9a207 (127.10.238.195:46121), f60a102acfbd4f6d9057d66c7d30f3fa (127.10.238.194:44521)
I20260929 10:56:17.565858 22844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbd380524db4aab970de4809686f7eb" candidate_uuid: "032ef7ab73944bdd863a7733f0b45627" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c4fa7db2ba482d89a6d60cb5b9a207"
I20260929 10:56:17.565949 22844 raft_consensus.cc:3037] T 6dbd380524db4aab970de4809686f7eb P 84c4fa7db2ba482d89a6d60cb5b9a207 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:56:17.566833 22844 raft_consensus.cc:2454] T 6dbd380524db4aab970de4809686f7eb P 84c4fa7db2ba482d89a6d60cb5b9a207 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 032ef7ab73944bdd863a7733f0b45627 in term 1.
I20260929 10:56:17.567021 22505 leader_election.cc:302] T 6dbd380524db4aab970de4809686f7eb P 032ef7ab73944bdd863a7733f0b45627 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 032ef7ab73944bdd863a7733f0b45627, 84c4fa7db2ba482d89a6d60cb5b9a207; no voters:
I20260929 10:56:17.567121 22917 raft_consensus.cc:2781] T 6dbd380524db4aab970de4809686f7eb P 032ef7ab73944bdd863a7733f0b45627 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:56:17.567269 22917 raft_consensus.cc:686] T 6dbd380524db4aab970de4809686f7eb P 032ef7ab73944bdd863a7733f0b45627 [term 1 LEADER]: Becoming Leader. State: Replica: 032ef7ab73944bdd863a7733f0b45627, State: Running, Role: LEADER
I20260929 10:56:17.567382 22917 consensus_queue.cc:237] T 6dbd380524db4aab970de4809686f7eb P 032ef7ab73944bdd863a7733f0b45627 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c4fa7db2ba482d89a6d60cb5b9a207" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 46121 } } peers { permanent_uuid: "f60a102acfbd4f6d9057d66c7d30f3fa" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 44521 } } peers { permanent_uuid: "032ef7ab73944bdd863a7733f0b45627" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 37123 } }
I20260929 10:56:17.568250 22407 catalog_manager.cc:5495] T 6dbd380524db4aab970de4809686f7eb P 032ef7ab73944bdd863a7733f0b45627 reported cstate change: term changed from 0 to 1, leader changed from <none> to 032ef7ab73944bdd863a7733f0b45627 (127.10.238.193). New cstate: current_term: 1 leader_uuid: "032ef7ab73944bdd863a7733f0b45627" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c4fa7db2ba482d89a6d60cb5b9a207" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 46121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f60a102acfbd4f6d9057d66c7d30f3fa" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 44521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "032ef7ab73944bdd863a7733f0b45627" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 37123 } health_report { overall_health: HEALTHY } } }
I20260929 10:56:17.570554 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:56:17.550507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:43243 (local address 127.10.238.194:44521)
0929 10:56:17.550545 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 10:56:17.550547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:56:17.552434 (+ 1887us) server_negotiation.cc:407] Connection header received
0929 10:56:17.552498 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:56:17.552499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:56:17.552511 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:56:17.552571 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:56:17.552722 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:56:17.553413 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:56:17.569591 (+ 16178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:56:17.570134 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:56:17.570191 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:56:17.570201 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 10:56:17.570438 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 10:56:17.570486 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 10:56:17.570780 22920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:56:17.550299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.194:44521 (local address 127.10.238.193:43243)
0929 10:56:17.552361 (+ 2062us) negotiation.cc:107] Waiting for socket to connect
0929 10:56:17.552374 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:56:17.552444 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:56:17.552589 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:56:17.552589 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:56:17.552591 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:56:17.552666 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:56:17.552668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:56:17.567858 (+ 15190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:56:17.567860 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:56:17.568969 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:56:17.568973 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:56:17.570264 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:56:17.570265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:56:17.570338 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:56:17.570355 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 10:56:17.570644 (+ 289us) client_negotiation.cc:240] Negotiation successful
0929 10:56:17.570693 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2031,"thread_start_us":76,"threads_started":1}
I20260929 10:56:17.571205 22711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbd380524db4aab970de4809686f7eb" candidate_uuid: "032ef7ab73944bdd863a7733f0b45627" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f60a102acfbd4f6d9057d66c7d30f3fa" is_pre_election: true
I20260929 10:56:17.571394 22711 raft_consensus.cc:2454] T 6dbd380524db4aab970de4809686f7eb P f60a102acfbd4f6d9057d66c7d30f3fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 032ef7ab73944bdd863a7733f0b45627 in term 0.
I20260929 10:56:17.571210 22710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbd380524db4aab970de4809686f7eb" candidate_uuid: "032ef7ab73944bdd863a7733f0b45627" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f60a102acfbd4f6d9057d66c7d30f3fa"
I20260929 10:56:17.571524 22710 raft_consensus.cc:3037] T 6dbd380524db4aab970de4809686f7eb P f60a102acfbd4f6d9057d66c7d30f3fa [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:56:17.572328 22710 raft_consensus.cc:2454] T 6dbd380524db4aab970de4809686f7eb P f60a102acfbd4f6d9057d66c7d30f3fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 032ef7ab73944bdd863a7733f0b45627 in term 1.
I20260929 10:56:17.575120 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:56:17.558240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.194:44521 (local address 127.10.238.195:42569)
0929 10:56:17.558358 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 10:56:17.558368 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 10:56:17.558407 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:56:17.569963 (+ 11556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:56:17.569965 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:56:17.569967 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:56:17.570029 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:56:17.570031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:56:17.572849 (+ 2818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:56:17.572850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:56:17.573982 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:56:17.573984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:56:17.574917 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:56:17.574918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:56:17.574985 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:56:17.574990 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 10:56:17.575017 (+ 27us) client_negotiation.cc:240] Negotiation successful
0929 10:56:17.575056 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20260929 10:56:17.576715 22922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:56:17.558479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:42569 (local address 127.10.238.194:44521)
0929 10:56:17.569816 (+ 11337us) server_negotiation.cc:206] Beginning negotiation
0929 10:56:17.569818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:56:17.569828 (+ 10us) server_negotiation.cc:407] Connection header received
0929 10:56:17.569852 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:56:17.569853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:56:17.569863 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:56:17.570086 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:56:17.570122 (+ 36us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:56:17.570920 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:56:17.574136 (+ 3216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:56:17.574825 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:56:17.576588 (+ 1763us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:56:17.576600 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0929 10:56:17.576617 (+ 17us) server_negotiation.cc:299] Negotiation successful
0929 10:56:17.576656 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11313,"thread_start_us":51,"threads_started":1}
I20260929 10:56:17.576967 22710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbd380524db4aab970de4809686f7eb" candidate_uuid: "84c4fa7db2ba482d89a6d60cb5b9a207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f60a102acfbd4f6d9057d66c7d30f3fa" is_pre_election: true
I20260929 10:56:17.577049 22710 raft_consensus.cc:2379] T 6dbd380524db4aab970de4809686f7eb P f60a102acfbd4f6d9057d66c7d30f3fa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 84c4fa7db2ba482d89a6d60cb5b9a207 in current term 1: Already voted for candidate 032ef7ab73944bdd863a7733f0b45627 in this term.
I20260929 10:56:17.577703 22923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:56:17.558691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:37123 (local address 127.10.238.195:35993)
0929 10:56:17.572695 (+ 14004us) negotiation.cc:107] Waiting for socket to connect
0929 10:56:17.572706 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 10:56:17.572761 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:56:17.574218 (+ 1457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:56:17.574219 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:56:17.574222 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:56:17.574276 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:56:17.574277 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:56:17.575405 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:56:17.575406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:56:17.576470 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:56:17.576473 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:56:17.577466 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:56:17.577467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:56:17.577530 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:56:17.577534 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 10:56:17.577586 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 10:56:17.577620 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13983,"thread_start_us":87,"threads_started":1}
I20260929 10:56:17.577718 22906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:56:17.558746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:35993 (local address 127.10.238.193:37123)
0929 10:56:17.558785 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 10:56:17.558803 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 10:56:17.572754 (+ 13951us) server_negotiation.cc:407] Connection header received
0929 10:56:17.572806 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:56:17.572807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:56:17.572816 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:56:17.572877 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:56:17.574337 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:56:17.574960 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:56:17.576547 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:56:17.577015 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:56:17.577086 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:56:17.577094 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 10:56:17.577608 (+ 514us) server_negotiation.cc:299] Negotiation successful
0929 10:56:17.577658 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 10:56:17.578223 22571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbd380524db4aab970de4809686f7eb" candidate_uuid: "84c4fa7db2ba482d89a6d60cb5b9a207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "032ef7ab73944bdd863a7733f0b45627" is_pre_election: true
I20260929 10:56:17.578558 22781 leader_election.cc:302] T 6dbd380524db4aab970de4809686f7eb P 84c4fa7db2ba482d89a6d60cb5b9a207 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84c4fa7db2ba482d89a6d60cb5b9a207; no voters: 032ef7ab73944bdd863a7733f0b45627, f60a102acfbd4f6d9057d66c7d30f3fa
I20260929 10:56:17.578715 22915 raft_consensus.cc:2726] T 6dbd380524db4aab970de4809686f7eb P 84c4fa7db2ba482d89a6d60cb5b9a207 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 10:56:17.596177 22906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:56:17.593679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57578 (local address 127.10.238.193:37123)
0929 10:56:17.593719 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 10:56:17.593721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:56:17.593728 (+ 7us) server_negotiation.cc:407] Connection header received
0929 10:56:17.593763 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:56:17.593764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:56:17.593783 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:56:17.593862 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:56:17.594068 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:56:17.594667 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:56:17.595408 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:56:17.595592 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:56:17.595661 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:56:17.595850 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:56:17.595945 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:56:17.596035 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 10:56:17.596054 (+ 19us) server_negotiation.cc:299] Negotiation successful
0929 10:56:17.596072 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 10:56:17.598161 22710 raft_consensus.cc:1261] T 6dbd380524db4aab970de4809686f7eb P f60a102acfbd4f6d9057d66c7d30f3fa [term 1 FOLLOWER]: Refusing update from remote peer 032ef7ab73944bdd863a7733f0b45627: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:56:17.598177 22844 raft_consensus.cc:1261] T 6dbd380524db4aab970de4809686f7eb P 84c4fa7db2ba482d89a6d60cb5b9a207 [term 1 FOLLOWER]: Refusing update from remote peer 032ef7ab73944bdd863a7733f0b45627: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:56:17.598495 22917 consensus_queue.cc:1035] T 6dbd380524db4aab970de4809686f7eb P 032ef7ab73944bdd863a7733f0b45627 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c4fa7db2ba482d89a6d60cb5b9a207" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 46121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:56:17.598683 22917 consensus_queue.cc:1035] T 6dbd380524db4aab970de4809686f7eb P 032ef7ab73944bdd863a7733f0b45627 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f60a102acfbd4f6d9057d66c7d30f3fa" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 44521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:56:17.601691 22928 mvcc.cc:205] Tried to move back new op lower bound from 7334622730638929920 to 7334622730516549632. Current Snapshot: MvccSnapshot[applied={T|T < 7334622730638929920}]
I20260929 10:56:17.605764 22406 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:56:17.605870 22406 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:56:17.606701 22406 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 502db2680c6a4fe5af2c94049e38faad: Sending DeleteTablet for 3 replicas of tablet 6dbd380524db4aab970de4809686f7eb
I20260929 10:56:17.607023 22542 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dbd380524db4aab970de4809686f7eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:56:17 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:57564
I20260929 10:56:17.607162 22677 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dbd380524db4aab970de4809686f7eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:56:17 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:53488
I20260929 10:56:17.607236 22541 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dbd380524db4aab970de4809686f7eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:56:17 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:57564
I20260929 10:56:17.607277 22938 tablet_replica.cc:330] T 6dbd380524db4aab970de4809686f7eb P 032ef7ab73944bdd863a7733f0b45627: stopping tablet replica
I20260929 10:56:17.607384 22938 raft_consensus.cc:2229] T 6dbd380524db4aab970de4809686f7eb P 032ef7ab73944bdd863a7733f0b45627 [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:56:17.607501 22938 raft_consensus.cc:2258] T 6dbd380524db4aab970de4809686f7eb P 032ef7ab73944bdd863a7733f0b45627 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:56:17.607770 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 22488
I20260929 10:56:17.607947 22938 ts_tablet_manager.cc:1905] T 6dbd380524db4aab970de4809686f7eb P 032ef7ab73944bdd863a7733f0b45627: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:56:17.610685 22929 mvcc.cc:205] Tried to move back new op lower bound from 7334622730638929920 to 7334622730516549632. Current Snapshot: MvccSnapshot[applied={T|T < 7334622730638929920 or (T in {7334622730638929920})}]
I20260929 10:56:17.611002 22825 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dbd380524db4aab970de4809686f7eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:56:17 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:43322
I20260929 10:56:17.611267 22931 mvcc.cc:205] Tried to move back new op lower bound from 7334622730638929920 to 7334622730516549632. Current Snapshot: MvccSnapshot[applied={T|T < 7334622730638929920 or (T in {7334622730638929920})}]
I20260929 10:56:17.614856 22939 tablet_replica.cc:330] T 6dbd380524db4aab970de4809686f7eb P f60a102acfbd4f6d9057d66c7d30f3fa: stopping tablet replica
I20260929 10:56:17.614981 22939 raft_consensus.cc:2229] T 6dbd380524db4aab970de4809686f7eb P f60a102acfbd4f6d9057d66c7d30f3fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:56:17.615099 22939 raft_consensus.cc:2258] T 6dbd380524db4aab970de4809686f7eb P f60a102acfbd4f6d9057d66c7d30f3fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:56:17.616807 22939 ts_tablet_manager.cc:1905] T 6dbd380524db4aab970de4809686f7eb P f60a102acfbd4f6d9057d66c7d30f3fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:56:17.620914 22939 ts_tablet_manager.cc:1918] T 6dbd380524db4aab970de4809686f7eb P f60a102acfbd4f6d9057d66c7d30f3fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:56:17.621826 22939 log.cc:1196] T 6dbd380524db4aab970de4809686f7eb P f60a102acfbd4f6d9057d66c7d30f3fa: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790679274510869-11195-0/minicluster-data/ts-1/wal/wals/6dbd380524db4aab970de4809686f7eb
W20260929 10:56:17.622875 22394 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.238.193:37123 (error 108)
I20260929 10:56:17.623301 22939 ts_tablet_manager.cc:1939] T 6dbd380524db4aab970de4809686f7eb P f60a102acfbd4f6d9057d66c7d30f3fa: Deleting consensus metadata
I20260929 10:56:17.624315 22393 catalog_manager.cc:4841] TS f60a102acfbd4f6d9057d66c7d30f3fa (127.10.238.194:44521): tablet 6dbd380524db4aab970de4809686f7eb (table test-table [id=1cbaf9c2b3b94ecab68984a56287431e]) successfully deleted
W20260929 10:56:17.624315 22392 connection.cc:535] server connection from 127.10.238.193:48351 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 10:56:17.625432 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 22625
I20260929 10:56:17.626338 22940 tablet_replica.cc:330] T 6dbd380524db4aab970de4809686f7eb P 84c4fa7db2ba482d89a6d60cb5b9a207: stopping tablet replica
I20260929 10:56:17.626531 22940 raft_consensus.cc:2229] T 6dbd380524db4aab970de4809686f7eb P 84c4fa7db2ba482d89a6d60cb5b9a207 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:56:17.626696 22940 raft_consensus.cc:2258] T 6dbd380524db4aab970de4809686f7eb P 84c4fa7db2ba482d89a6d60cb5b9a207 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:56:17.630893 22905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:56:17.630754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:37123 (local address 127.0.0.1:57592)
0929 10:56:17.630818 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0929 10:56:17.630858 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.193:37123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 10:56:17.631264 22394 catalog_manager.cc:4584] TS 032ef7ab73944bdd863a7733f0b45627 (127.10.238.193:37123): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6dbd380524db4aab970de4809686f7eb: Network error: Client connection negotiation failed: client connection to 127.10.238.193:37123: connect: Connection refused (error 111)
I20260929 10:56:17.632205 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 22762
I20260929 10:56:17.639020 22905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:56:17.638901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.195:46121 (local address 127.0.0.1:43338)
0929 10:56:17.638954 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0929 10:56:17.638980 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.195:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 10:56:17.639250 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 22375
I20260929 10:56:17.654664 11195 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 10:56:18.000013 11195 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 10:56:18.000154 11195 mini_ranger.cc:66] Stopping Ranger...
I20260929 10:56:18.345675 11195 mini_ranger.cc:68] Stopped Ranger
2026-09-29 10:56:18.345 UTC [21960] LOG: received smart shutdown request
2026-09-29 10:56:18.349 UTC [21960] LOG: background worker "logical replication launcher" (PID 21969) exited with exit code 1
2026-09-29 10:56:18.349 UTC [21964] LOG: shutting down
2026-09-29 10:56:18.526 UTC [21960] LOG: database system is shut down
2026-09-29T10:56:18Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49144 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:56:18 dist-test-slave-8nrt krb5kdc[22950](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:56:18 dist-test-slave-8nrt krb5kdc[22950](info): set up 2 sockets
Sep 29 10:56:18 dist-test-slave-8nrt krb5kdc[22950](info): commencing operation
krb5kdc: starting...
W20260929 10:56:19.673213 11195 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Sep 29 10:56:19 dist-test-slave-8nrt krb5kdc[22950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:56:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:56:19Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.10.238.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.10.238.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.10.238.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/rangeradmin_127.10.238.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.10.238.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/rangeradmin_127.10.238.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.10.238.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.10.238.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.10.238.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/rangerlookup_127.10.238.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.10.238.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/rangerlookup_127.10.238.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.238.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.238.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.238.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/HTTP_127.10.238.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.238.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/HTTP_127.10.238.212@KRBTEST.COM.keytab.
I20260929 10:56:19.827414 11195 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskc60rDW/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/postgres -l logfile start
2026-09-29 10:56:22.224 UTC [22980] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 10:56:22.224 UTC [22980] LOG: listening on IPv4 address "127.10.238.212", port 48471
2026-09-29 10:56:22.227 UTC [22980] LOG: listening on Unix socket "/tmp/.s.PGSQL.48471"
2026-09-29 10:56:22.250 UTC [22983] LOG: database system was shut down at 2026-09-29 10:56:20 UTC
2026-09-29 10:56:22.254 UTC [22980] LOG: database system is ready to accept connections
I20260929 10:56:23.205195 11195 mini_postgres.cc:96] Postgres bound to 48471
2026-09-29 10:56:23.211 UTC [22991] FATAL: database "slave" does not exist
127.10.238.212:48471 - accepting connections
I20260929 10:56:23.211815 11195 mini_ranger.cc:161] Starting Ranger...
I20260929 10:56:23.226707 11195 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 10:56:23.703876 11195 mini_ranger.cc:87] Created ranger Postgres database
I20260929 10:56:23.703965 11195 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ranger-admin
2026-09-29 10:56:23,998 [I] DB FLAVOR :POSTGRES
2026-09-29 10:56:23,999 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 10:56:23,999 [I] Checking connection..
2026-09-29 10:56:23,999 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc60rDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskc60rDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.238.212:48471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 10:56:24,265 [I] Checking connection passed.
2026-09-29 10:56:24,266 [I] --------- Verifying version history table ---------
2026-09-29 10:56:24,266 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc60rDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskc60rDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.238.212:48471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 10:56:24,502 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 10:56:24,503 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc60rDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskc60rDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.238.212:48471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 10:56:24,763 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 10:56:24,763 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 10:56:24,763 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc60rDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskc60rDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.238.212:48471/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskc60rDW/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 10:56:25.026 UTC [23055] WARNING: there is no transaction in progress
2026-09-29 10:56:25,033 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 10:56:25,033 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc60rDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskc60rDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.238.212:48471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 10:56:25,266 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 10:56:25,267 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 10:56:25,267 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc60rDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskc60rDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.238.212:48471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 10:56:25,503 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc60rDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskc60rDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.238.212:48471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 10:56:25,759 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc60rDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskc60rDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.238.212:48471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-8nrt.c.gcp-upstream.internal','N') ;"
2026-09-29 10:56:26,014 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc60rDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskc60rDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.238.212:48471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 10:56:26,259 [I] Table x_portal_user does not exist in database ranger
2026-09-29 10:56:26,260 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc60rDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskc60rDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.238.212:48471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 10:56:26,518 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 10:56:26,519 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc60rDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskc60rDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.238.212:48471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 10:56:26,767 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc60rDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskc60rDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.238.212:48471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 10:56:27,025 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 10:56:27,025 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc60rDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskc60rDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.238.212:48471/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskc60rDW/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 10:56:27.973 UTC [23163] WARNING: there is no transaction in progress
2026-09-29 10:56:27.988 UTC [23163] WARNING: there is no transaction in progress
2026-09-29 10:56:28.005 UTC [23163] WARNING: there is no transaction in progress
2026-09-29 10:56:28.020 UTC [23163] WARNING: there is no transaction in progress
2026-09-29 10:56:28.035 UTC [23163] WARNING: there is no transaction in progress
2026-09-29 10:56:28.052 UTC [23163] WARNING: there is no transaction in progress
2026-09-29 10:56:28.117 UTC [23163] WARNING: there is no transaction in progress
2026-09-29 10:56:28.123 UTC [23163] WARNING: there is no transaction in progress
2026-09-29 10:56:28.137 UTC [23163] WARNING: there is no transaction in progress
2026-09-29 10:56:28.147 UTC [23163] WARNING: there is no transaction in progress
2026-09-29 10:56:28.157 UTC [23163] WARNING: there is no transaction in progress
2026-09-29 10:56:28.171 UTC [23163] WARNING: there is no transaction in progress
2026-09-29 10:56:28.181 UTC [23163] WARNING: there is no transaction in progress
2026-09-29 10:56:28.188 UTC [23163] WARNING: there is no transaction in progress
2026-09-29 10:56:28.194 UTC [23163] WARNING: there is no transaction in progress
2026-09-29 10:56:28.200 UTC [23163] WARNING: there is no transaction in progress
2026-09-29 10:56:28,810 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 10:56:28,810 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc60rDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskc60rDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.238.212:48471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 10:56:29,055 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 10:56:29,056 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc60rDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskc60rDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.238.212:48471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 10:56:29,299 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 10:56:29,299 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc60rDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskc60rDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.238.212:48471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 10:56:29,568 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 10:56:29,568 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc60rDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskc60rDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.238.212:48471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 10:56:29,823 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc60rDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskc60rDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.238.212:48471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 10:56:30,088 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc60rDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskc60rDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.238.212:48471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-8nrt.c.gcp-upstream.internal';"
2026-09-29 10:56:30,323 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 10:56:30,324 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc60rDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskc60rDW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.238.212:48471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 10:56:30,572 [I] DB_PATCHES have already been applied
I20260929 10:56:30.577927 11195 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ranger-admin:/tmp/dist-test-taskc60rDW/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskc60rDW/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskc60rDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskc60rDW/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 10:56:30.577987 11195 mini_ranger.cc:193] Using host: 127.10.238.212
I20260929 10:56:30.580979 11195 mini_ranger.cc:227] Ranger admin URL: http://127.10.238.212:33405
Sep 29, 2026 10:56:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 10:56:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 10:56:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 10:56:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 10:56:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 10:56:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 10:56:31 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 10:56:31.693943 11195 mini_ranger.cc:160] Time spent starting Ranger: real 8.482s user 0.000s sys 0.009s
Sep 29, 2026 10:56:51 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 10:56:51 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 10:56:51 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 10:56:51 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 10:56:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 10:56:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 10:56:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 10:56:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 10:56:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 10:56:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 10:56:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 10:56:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 10:56:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 10:56:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 10:56:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 10:56:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 10:56:51 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 10:56:52 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 10:56:53.076758 11195 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.10.238.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.10.238.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.10.238.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/rangerkms_127.10.238.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.10.238.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/rangerkms_127.10.238.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.238.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.10.238.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.238.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/HTTP_127.10.238.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.238.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/HTTP_127.10.238.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 10:56:53 dist-test-slave-8nrt krb5kdc[22950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679413, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 10:56:53.161923 11195 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 10:56:53.171281 11195 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 10:56:54.003598 11195 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 10:56:54.003685 11195 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ranger-kms
I20260929 10:56:54.003707 11195 mini_ranger_kms.cc:227] Using postgres at 127.10.238.212:48471
2026-09-29 10:56:54,171 [I] DB FLAVOR :POSTGRES
2026-09-29 10:56:54,171 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 10:56:54,171 [I] Checking connection
2026-09-29 10:56:54,171 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc60rDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.238.212:48471/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 10:56:54,437 [I] connection success
2026-09-29 10:56:54,438 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 10:56:54,438 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 10:56:54,438 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc60rDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.238.212:48471/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 10:56:54,667 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 10:56:54,667 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 10:56:54,668 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 10:56:54,668 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc60rDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.238.212:48471/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 10:56:54,941 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 10:56:55.438858 11195 mini_ranger_kms.cc:310] Created kms service
I20260929 10:56:55.647943 11195 mini_ranger_kms.cc:326] Created kudu user
I20260929 10:56:55.693746 11195 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 10:56:55 dist-test-slave-8nrt krb5kdc[22950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679413, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.10.238.212@KRBTEST.COM
I20260929 10:56:56.088989 11195 mini_ranger_kms.cc:384] Added ranger policy
I20260929 10:56:56.089200 11195 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ranger-kms:/tmp/dist-test-taskc60rDW/build/release/bin/postgresql.jar:/tmp/dist-test-taskc60rDW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskc60rDW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 10:56:56.089224 11195 mini_ranger_kms.cc:240] Using host: 127.10.238.212
I20260929 10:56:56.092700 11195 mini_ranger_kms.cc:276] Ranger KMS PID: 23343
I20260929 10:56:56.092775 11195 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.10.238.212:35939
Sep 29, 2026 10:56:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 10:56:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 10:56:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 10:56:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 10:56:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 10:56:57 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 10:56:58.311029 11195 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.149s user 0.008s sys 0.142s
I20260929 10:56:58.311133 11195 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 10:56:58.311185 11195 mini_ranger_kms.cc:401] 127.10.238.212:35939/kms/v1/keys
Sep 29 10:56:58 dist-test-slave-8nrt krb5kdc[22950](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790679418, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.238.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 10:56:58 dist-test-slave-8nrt krb5kdc[22950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679418, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.238.212@KRBTEST.COM for HTTP/127.10.238.212@KRBTEST.COM
Sep 29 10:56:59 dist-test-slave-8nrt krb5kdc[22950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679418, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.238.212@KRBTEST.COM for HTTP/127.10.238.212@KRBTEST.COM
Sep 29 10:57:01 dist-test-slave-8nrt krb5kdc[22950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.10.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
I20260929 10:57:01.566345 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:34825
--webserver_interface=127.10.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:35359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:34825
--ranger_config_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.238.212:35939/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:57:01.644665 23394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:57:01.644898 23394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:57:01.644936 23394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:57:01.646158 23394 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:57:01.646222 23394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:57:01.646246 23394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:57:01.646272 23394 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:57:01.646303 23394 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:57:01.647835 23394 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:35359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/master-0/wal
--ranger_kms_url=127.10.238.212:35939/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:34825
--ranger_config_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:34825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:57:01.648118 23394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:57:01.648365 23394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:57:01.654066 23399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:01.654072 23400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:01.654078 23402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:57:01.654942 23394 server_base.cc:1034] running on GCE node
I20260929 10:57:01.655267 23394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:57:01.655598 23394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:57:01.656786 23394 hybrid_clock.cc:648] HybridClock initialized: now 1790679421656776 us; error 40 us; skew 500 ppm
Sep 29 10:57:01 dist-test-slave-8nrt krb5kdc[22950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679421, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:57:01.662884 23394 init.cc:377] Logged in from keytab as kudu/127.10.238.254@KRBTEST.COM (short username kudu)
I20260929 10:57:01.663372 23394 webserver.cc:460] Webserver started at http://127.10.238.254:35647/ using document root <none> and password file <none>
I20260929 10:57:01.663610 23394 fs_manager.cc:359] Metadata directory not provided
I20260929 10:57:01.663669 23394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:57:01.663839 23394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 10:57:01 dist-test-slave-8nrt krb5kdc[22950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679421, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for HTTP/127.10.238.212@KRBTEST.COM
I20260929 10:57:01.716538 23394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/master-0/data/instance:
uuid: "5011a7bab2194a5985afd5dd97527e21"
format_stamp: "Formatted at 2026-09-29 10:57:01 on dist-test-slave-8nrt"
server_key: "283d7e10f632945e9dc50091bf337940"
server_key_iv: "604101599402b38745cb28a330baa95b"
server_key_version: "kuduclusterkey@0"
I20260929 10:57:01.716954 23394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/master-0/wal/instance:
uuid: "5011a7bab2194a5985afd5dd97527e21"
format_stamp: "Formatted at 2026-09-29 10:57:01 on dist-test-slave-8nrt"
server_key: "283d7e10f632945e9dc50091bf337940"
server_key_iv: "604101599402b38745cb28a330baa95b"
server_key_version: "kuduclusterkey@0"
I20260929 10:57:01.718485 23394 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260929 10:57:01.730295 23409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:01.730538 23394 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 10:57:01.730664 23394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "5011a7bab2194a5985afd5dd97527e21"
format_stamp: "Formatted at 2026-09-29 10:57:01 on dist-test-slave-8nrt"
server_key: "283d7e10f632945e9dc50091bf337940"
server_key_iv: "604101599402b38745cb28a330baa95b"
server_key_version: "kuduclusterkey@0"
I20260929 10:57:01.730777 23394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:57:01.758177 23394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:57:01.760293 23394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:57:01.760464 23394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:57:01.765226 23394 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.254:34825
I20260929 10:57:01.765308 23461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.254:34825 every 8 connection(s)
I20260929 10:57:01.765650 23394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:57:01.766426 23462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:57:01.769377 23462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5011a7bab2194a5985afd5dd97527e21: Bootstrap starting.
I20260929 10:57:01.770037 23462 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5011a7bab2194a5985afd5dd97527e21: Neither blocks nor log segments found. Creating new log.
I20260929 10:57:01.770349 23462 log.cc:824] T 00000000000000000000000000000000 P 5011a7bab2194a5985afd5dd97527e21: Log is configured to *not* fsync() on all Append() calls
I20260929 10:57:01.771085 23462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5011a7bab2194a5985afd5dd97527e21: No bootstrap required, opened a new log
I20260929 10:57:01.771567 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 23394
I20260929 10:57:01.771679 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/master-0/wal/instance
I20260929 10:57:01.771865 11195 external_mini_cluster.cc:1428] Setting key 0217543adc18be74b7ef2abb9519536a
I20260929 10:57:01.773747 23462 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5011a7bab2194a5985afd5dd97527e21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5011a7bab2194a5985afd5dd97527e21" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 34825 } }
I20260929 10:57:01.773993 23462 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5011a7bab2194a5985afd5dd97527e21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:57:01.774116 23462 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5011a7bab2194a5985afd5dd97527e21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5011a7bab2194a5985afd5dd97527e21, State: Initialized, Role: FOLLOWER
I20260929 10:57:01.774307 23462 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5011a7bab2194a5985afd5dd97527e21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5011a7bab2194a5985afd5dd97527e21" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 34825 } }
I20260929 10:57:01.774473 23462 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5011a7bab2194a5985afd5dd97527e21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:57:01.774552 23462 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5011a7bab2194a5985afd5dd97527e21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:57:01.774610 23462 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5011a7bab2194a5985afd5dd97527e21 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:57:01.775296 23462 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5011a7bab2194a5985afd5dd97527e21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5011a7bab2194a5985afd5dd97527e21" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 34825 } }
I20260929 10:57:01.775404 23462 leader_election.cc:302] T 00000000000000000000000000000000 P 5011a7bab2194a5985afd5dd97527e21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5011a7bab2194a5985afd5dd97527e21; no voters:
I20260929 10:57:01.775593 23462 leader_election.cc:288] T 00000000000000000000000000000000 P 5011a7bab2194a5985afd5dd97527e21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:57:01.775651 23467 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5011a7bab2194a5985afd5dd97527e21 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:57:01.775828 23467 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5011a7bab2194a5985afd5dd97527e21 [term 1 LEADER]: Becoming Leader. State: Replica: 5011a7bab2194a5985afd5dd97527e21, State: Running, Role: LEADER
I20260929 10:57:01.775920 23462 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5011a7bab2194a5985afd5dd97527e21 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:57:01.776015 23467 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5011a7bab2194a5985afd5dd97527e21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5011a7bab2194a5985afd5dd97527e21" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 34825 } }
I20260929 10:57:01.776434 23467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5011a7bab2194a5985afd5dd97527e21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5011a7bab2194a5985afd5dd97527e21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5011a7bab2194a5985afd5dd97527e21" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 34825 } } }
I20260929 10:57:01.776563 23467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5011a7bab2194a5985afd5dd97527e21 [sys.catalog]: This master's current role is: LEADER
I20260929 10:57:01.776732 23467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5011a7bab2194a5985afd5dd97527e21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5011a7bab2194a5985afd5dd97527e21. Latest consensus state: current_term: 1 leader_uuid: "5011a7bab2194a5985afd5dd97527e21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5011a7bab2194a5985afd5dd97527e21" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 34825 } } }
I20260929 10:57:01.776937 23467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5011a7bab2194a5985afd5dd97527e21 [sys.catalog]: This master's current role is: LEADER
I20260929 10:57:01.777320 23462 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Sep 29 10:57:01 dist-test-slave-8nrt krb5kdc[22950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:57:01.781507 23476 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:57:01.782091 23476 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:57:01.784132 23476 catalog_manager.cc:1347] Generated new cluster ID: 470f7ba3816549969eae461f4a9a4d55
I20260929 10:57:01.784174 23476 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:57:01.792542 23466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:01.772356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33790 (local address 127.10.238.254:34825)
0929 10:57:01.772552 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:01.772562 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:01.772637 (+ 75us) server_negotiation.cc:407] Connection header received
0929 10:57:01.773650 (+ 1013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:01.773658 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:01.773708 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:01.773865 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:57:01.774564 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:01.775624 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:01.776763 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:01.777443 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:01.777555 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:01.780294 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:57:01.780344 (+ 50us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:57:01.780345 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:57:01.780417 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:57:01.790348 (+ 9931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:01.790832 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:01.790835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:01.790835 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:01.790940 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:01.791164 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:01.791164 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:01.791165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:01.791484 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:57:01.792092 (+ 608us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:01.792165 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 10:57:01.792319 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":94,"threads_started":1}
I20260929 10:57:01.799283 23476 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:57:01.800009 23476 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:57:01.803472 23476 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5011a7bab2194a5985afd5dd97527e21: Generated new TSK 0
I20260929 10:57:01.803691 23476 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.238.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab.
Entry for principal kudu/127.10.238.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab.
I20260929 10:57:02.562461 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.238.212:35939/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.193:0
--local_ip_for_outbound_sockets=127.10.238.193
--webserver_interface=127.10.238.193
--webserver_port=0
--tserver_master_addrs=127.10.238.254:34825
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab
--principal=kudu/127.10.238.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:35359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:57:02.674247 23507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:57:02.674479 23507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:57:02.674520 23507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:57:02.675897 23507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:57:02.675956 23507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:57:02.676000 23507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.193
I20260929 10:57:02.677805 23507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:35359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.10.238.212:35939/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab
--principal=kudu/127.10.238.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.238.193
--webserver_port=0
--tserver_master_addrs=127.10.238.254:34825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.193
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:57:02.678102 23507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:57:02.678351 23507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:57:02.683303 23512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:02.683497 23515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:02.683655 23513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:57:02.684046 23507 server_base.cc:1034] running on GCE node
I20260929 10:57:02.684268 23507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:57:02.684520 23507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:57:02.685676 23507 hybrid_clock.cc:648] HybridClock initialized: now 1790679422685657 us; error 43 us; skew 500 ppm
Sep 29 10:57:02 dist-test-slave-8nrt krb5kdc[22950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679422, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:57:02.688449 23507 init.cc:377] Logged in from keytab as kudu/127.10.238.193@KRBTEST.COM (short username kudu)
I20260929 10:57:02.688953 23507 webserver.cc:460] Webserver started at http://127.10.238.193:41163/ using document root <none> and password file <none>
I20260929 10:57:02.689200 23507 fs_manager.cc:359] Metadata directory not provided
I20260929 10:57:02.689281 23507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:57:02.689431 23507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 10:57:02 dist-test-slave-8nrt krb5kdc[22950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679422, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.193@KRBTEST.COM for HTTP/127.10.238.212@KRBTEST.COM
I20260929 10:57:02.703464 23507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-0/data/instance:
uuid: "133a79f08d534b7f8fea4cac98dfce44"
format_stamp: "Formatted at 2026-09-29 10:57:02 on dist-test-slave-8nrt"
server_key: "0e37fd5a0a2ff8105cf03d57c949a0c0"
server_key_iv: "a955a1d5a29353e18a61ce36075cc15d"
server_key_version: "kuduclusterkey@0"
I20260929 10:57:02.704030 23507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-0/wal/instance:
uuid: "133a79f08d534b7f8fea4cac98dfce44"
format_stamp: "Formatted at 2026-09-29 10:57:02 on dist-test-slave-8nrt"
server_key: "0e37fd5a0a2ff8105cf03d57c949a0c0"
server_key_iv: "a955a1d5a29353e18a61ce36075cc15d"
server_key_version: "kuduclusterkey@0"
I20260929 10:57:02.706116 23507 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 10:57:02.721032 23522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:02.721233 23507 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 10:57:02.721354 23507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-0/wal
uuid: "133a79f08d534b7f8fea4cac98dfce44"
format_stamp: "Formatted at 2026-09-29 10:57:02 on dist-test-slave-8nrt"
server_key: "0e37fd5a0a2ff8105cf03d57c949a0c0"
server_key_iv: "a955a1d5a29353e18a61ce36075cc15d"
server_key_version: "kuduclusterkey@0"
I20260929 10:57:02.721467 23507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:57:02.730333 23507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:57:02.732344 23507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:57:02.732493 23507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:57:02.732753 23507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:57:02.733107 23507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:57:02.733152 23507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:02.733179 23507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:57:02.733194 23507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:02.740479 23507 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.193:34175
I20260929 10:57:02.740551 23635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.193:34175 every 8 connection(s)
I20260929 10:57:02.740986 23507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
I20260929 10:57:02.751201 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 23507
I20260929 10:57:02.751338 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-0/wal/instance
I20260929 10:57:02.751539 11195 external_mini_cluster.cc:1428] Setting key 241dd7702005d23a76da177de3638aea
Sep 29 10:57:02 dist-test-slave-8nrt krb5kdc[22950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679422, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.193@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:57:02.756502 23639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:02.741926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:34825 (local address 127.10.238.193:45519)
0929 10:57:02.745477 (+ 3551us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:02.745536 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:02.746204 (+ 668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:02.746598 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:57:02.746603 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:57:02.746863 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:57:02.747135 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:02.747142 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:02.748213 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:02.748214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:02.749012 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:02.749014 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:02.749599 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:02.749601 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:02.749745 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:02.750407 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:57:02.750419 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:57:02.752714 (+ 2295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:57:02.754785 (+ 2071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:02.754792 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:02.754794 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:02.755092 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:02.755358 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:02.755360 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:02.755360 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:02.755503 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:02.755804 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:57:02.755810 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:57:02.755972 (+ 162us) client_negotiation.cc:769] Sending connection context
0929 10:57:02.756081 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 10:57:02.756250 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3503,"thread_start_us":81,"threads_started":1}
I20260929 10:57:02.756847 23638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:02.741926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:45519 (local address 127.10.238.254:34825)
0929 10:57:02.745393 (+ 3467us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:02.745396 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:02.746356 (+ 960us) server_negotiation.cc:407] Connection header received
0929 10:57:02.746389 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:02.746390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:02.746405 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:02.746471 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:57:02.747256 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:02.748104 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:02.749109 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:02.749429 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:02.749505 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:02.752837 (+ 3332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:57:02.752839 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:57:02.752839 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:57:02.752847 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:57:02.754691 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:02.755182 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:02.755183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:02.755184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:02.755266 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:02.755580 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:02.755581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:02.755582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:02.755716 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:57:02.755777 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:02.756644 (+ 867us) server_negotiation.cc:299] Negotiation successful
0929 10:57:02.756734 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3428,"thread_start_us":86,"threads_started":1}
I20260929 10:57:02.757329 23636 heartbeater.cc:347] Connected to a master server at 127.10.238.254:34825
I20260929 10:57:02.757463 23636 heartbeater.cc:464] Registering TS with master...
I20260929 10:57:02.757678 23636 heartbeater.cc:511] Master 127.10.238.254:34825 requested a full tablet report, sending...
I20260929 10:57:02.758301 23426 ts_manager.cc:194] Registered new tserver with Master: 133a79f08d534b7f8fea4cac98dfce44 (127.10.238.193:34175)
I20260929 10:57:02.758915 23426 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.193@KRBTEST.COM'} at 127.10.238.193:45519
WARNING: no policy specified for kudu/127.10.238.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab.
Entry for principal kudu/127.10.238.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab.
I20260929 10:57:02.784643 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.238.212:35939/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.194:0
--local_ip_for_outbound_sockets=127.10.238.194
--webserver_interface=127.10.238.194
--webserver_port=0
--tserver_master_addrs=127.10.238.254:34825
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab
--principal=kudu/127.10.238.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:35359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:57:02.900868 23644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:57:02.901147 23644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:57:02.901189 23644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:57:02.903117 23644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:57:02.903224 23644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:57:02.903280 23644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.194
I20260929 10:57:02.905392 23644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:35359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.10.238.212:35939/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab
--principal=kudu/127.10.238.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.238.194
--webserver_port=0
--tserver_master_addrs=127.10.238.254:34825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.194
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:57:02.905723 23644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:57:02.905985 23644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:57:02.912117 23654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:02.914268 23652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:02.914320 23651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:57:02.915414 23644 server_base.cc:1034] running on GCE node
I20260929 10:57:02.915643 23644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:57:02.915908 23644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:57:02.917088 23644 hybrid_clock.cc:648] HybridClock initialized: now 1790679422917061 us; error 40 us; skew 500 ppm
Sep 29 10:57:02 dist-test-slave-8nrt krb5kdc[22950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679422, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:57:02.920356 23644 init.cc:377] Logged in from keytab as kudu/127.10.238.194@KRBTEST.COM (short username kudu)
I20260929 10:57:02.920883 23644 webserver.cc:460] Webserver started at http://127.10.238.194:36111/ using document root <none> and password file <none>
I20260929 10:57:02.921167 23644 fs_manager.cc:359] Metadata directory not provided
I20260929 10:57:02.921276 23644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:57:02.921425 23644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 10:57:02 dist-test-slave-8nrt krb5kdc[22950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679422, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.194@KRBTEST.COM for HTTP/127.10.238.212@KRBTEST.COM
I20260929 10:57:02.944423 23644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-1/data/instance:
uuid: "bde3d177c7c84afea69a699e117771f8"
format_stamp: "Formatted at 2026-09-29 10:57:02 on dist-test-slave-8nrt"
server_key: "41048e4567266e5ad99556b655bcd955"
server_key_iv: "9bbb3986ace25ac86dd9e17ba138f39c"
server_key_version: "kuduclusterkey@0"
I20260929 10:57:02.944922 23644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-1/wal/instance:
uuid: "bde3d177c7c84afea69a699e117771f8"
format_stamp: "Formatted at 2026-09-29 10:57:02 on dist-test-slave-8nrt"
server_key: "41048e4567266e5ad99556b655bcd955"
server_key_iv: "9bbb3986ace25ac86dd9e17ba138f39c"
server_key_version: "kuduclusterkey@0"
I20260929 10:57:02.947680 23644 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
Sep 29 10:57:02 dist-test-slave-8nrt krb5kdc[22950](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790679422, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:57:02.972407 23661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:02.972679 23644 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260929 10:57:02.972769 23644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-1/wal
uuid: "bde3d177c7c84afea69a699e117771f8"
format_stamp: "Formatted at 2026-09-29 10:57:02 on dist-test-slave-8nrt"
server_key: "41048e4567266e5ad99556b655bcd955"
server_key_iv: "9bbb3986ace25ac86dd9e17ba138f39c"
server_key_version: "kuduclusterkey@0"
I20260929 10:57:02.972864 23644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:57:03.007131 23644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:57:03.009696 23644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:57:03.009892 23644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:57:03.010113 23644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:57:03.010506 23644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:57:03.010553 23644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:03.010587 23644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:57:03.010609 23644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:03.017223 23644 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.194:45691
I20260929 10:57:03.017549 23774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.194:45691 every 8 connection(s)
I20260929 10:57:03.017769 23644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
Sep 29 10:57:03 dist-test-slave-8nrt krb5kdc[22950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679422, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.194@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:57:03.027772 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 23644
I20260929 10:57:03.027892 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-1/wal/instance
I20260929 10:57:03.028079 11195 external_mini_cluster.cc:1428] Setting key 6b2ea46f4d0c4470f3bf7c9c7f96f37f
I20260929 10:57:03.029433 23638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:03.018875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:51711 (local address 127.10.238.254:34825)
0929 10:57:03.018938 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:03.018940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:03.019096 (+ 156us) server_negotiation.cc:407] Connection header received
0929 10:57:03.019796 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:03.019798 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:03.019811 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:03.019844 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:57:03.020833 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:03.021670 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:03.023190 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:03.023523 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:03.023620 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:03.026044 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:57:03.026046 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:57:03.026046 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:57:03.026069 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:57:03.027847 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:03.028319 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:03.028320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:03.028320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:03.028374 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:03.028618 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:03.028618 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:03.028619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:03.028738 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:57:03.028784 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:03.029151 (+ 367us) server_negotiation.cc:299] Negotiation successful
0929 10:57:03.029250 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260929 10:57:03.029506 23777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:03.018590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:34825 (local address 127.10.238.194:51711)
0929 10:57:03.018970 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:03.019014 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:03.019666 (+ 652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:03.020028 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:57:03.020032 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:57:03.020332 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:57:03.020596 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:03.020602 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:03.021826 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:03.021826 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:03.023079 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:03.023081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:03.023635 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:03.023636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:03.023745 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:03.024296 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:57:03.024303 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:57:03.025929 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:57:03.027940 (+ 2011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:03.027947 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:03.027949 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:03.028218 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:03.028439 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:03.028440 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:03.028440 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:03.028552 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:03.028829 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:57:03.028835 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:57:03.028999 (+ 164us) client_negotiation.cc:769] Sending connection context
0929 10:57:03.029103 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 10:57:03.029278 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":89,"threads_started":1}
I20260929 10:57:03.030290 23775 heartbeater.cc:347] Connected to a master server at 127.10.238.254:34825
I20260929 10:57:03.030393 23775 heartbeater.cc:464] Registering TS with master...
I20260929 10:57:03.030591 23775 heartbeater.cc:511] Master 127.10.238.254:34825 requested a full tablet report, sending...
I20260929 10:57:03.030995 23426 ts_manager.cc:194] Registered new tserver with Master: bde3d177c7c84afea69a699e117771f8 (127.10.238.194:45691)
I20260929 10:57:03.031373 23426 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.194@KRBTEST.COM'} at 127.10.238.194:51711
WARNING: no policy specified for kudu/127.10.238.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab.
Entry for principal kudu/127.10.238.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab.
I20260929 10:57:03.057102 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.238.212:35939/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.195:0
--local_ip_for_outbound_sockets=127.10.238.195
--webserver_interface=127.10.238.195
--webserver_port=0
--tserver_master_addrs=127.10.238.254:34825
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab
--principal=kudu/127.10.238.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:35359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:57:03.146718 23780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:57:03.147042 23780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:57:03.147105 23780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:57:03.148782 23780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:57:03.148869 23780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:57:03.148913 23780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.195
I20260929 10:57:03.150435 23780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:35359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.10.238.212:35939/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab
--principal=kudu/127.10.238.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.238.195
--webserver_port=0
--tserver_master_addrs=127.10.238.254:34825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.195
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:57:03.150701 23780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:57:03.150954 23780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:57:03.156132 23787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:03.156132 23789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:57:03.157294 23780 server_base.cc:1034] running on GCE node
W20260929 10:57:03.156141 23786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:57:03.159287 23780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:57:03.159574 23780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:57:03.162871 23780 hybrid_clock.cc:648] HybridClock initialized: now 1790679423162843 us; error 37 us; skew 500 ppm
Sep 29 10:57:03 dist-test-slave-8nrt krb5kdc[22950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679423, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:57:03.166397 23780 init.cc:377] Logged in from keytab as kudu/127.10.238.195@KRBTEST.COM (short username kudu)
I20260929 10:57:03.166932 23780 webserver.cc:460] Webserver started at http://127.10.238.195:40083/ using document root <none> and password file <none>
I20260929 10:57:03.167177 23780 fs_manager.cc:359] Metadata directory not provided
I20260929 10:57:03.167224 23780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:57:03.167347 23780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 10:57:03 dist-test-slave-8nrt krb5kdc[22950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679423, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.195@KRBTEST.COM for HTTP/127.10.238.212@KRBTEST.COM
I20260929 10:57:03.180440 23780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-2/data/instance:
uuid: "1e4c5281b9df4fc58b8cc13dda2ccb61"
format_stamp: "Formatted at 2026-09-29 10:57:03 on dist-test-slave-8nrt"
server_key: "96604afa1d6a9fcfcc76e8ece88ffae1"
server_key_iv: "57a7606968a669491241618ef39de5e7"
server_key_version: "kuduclusterkey@0"
I20260929 10:57:03.180894 23780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-2/wal/instance:
uuid: "1e4c5281b9df4fc58b8cc13dda2ccb61"
format_stamp: "Formatted at 2026-09-29 10:57:03 on dist-test-slave-8nrt"
server_key: "96604afa1d6a9fcfcc76e8ece88ffae1"
server_key_iv: "57a7606968a669491241618ef39de5e7"
server_key_version: "kuduclusterkey@0"
I20260929 10:57:03.184247 23780 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260929 10:57:03.200866 23796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:03.201138 23780 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260929 10:57:03.201297 23780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-2/wal
uuid: "1e4c5281b9df4fc58b8cc13dda2ccb61"
format_stamp: "Formatted at 2026-09-29 10:57:03 on dist-test-slave-8nrt"
server_key: "96604afa1d6a9fcfcc76e8ece88ffae1"
server_key_iv: "57a7606968a669491241618ef39de5e7"
server_key_version: "kuduclusterkey@0"
I20260929 10:57:03.201402 23780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:57:03.216607 23780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:57:03.219151 23780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:57:03.219359 23780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:57:03.219631 23780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:57:03.220027 23780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:57:03.220111 23780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:03.220180 23780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:57:03.220235 23780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:03.226410 23780 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.195:33633
I20260929 10:57:03.226904 23780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
I20260929 10:57:03.230451 23909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.195:33633 every 8 connection(s)
I20260929 10:57:03.235330 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 23780
I20260929 10:57:03.235464 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-2/wal/instance
I20260929 10:57:03.235716 11195 external_mini_cluster.cc:1428] Setting key bc4a60d03740b5e5e65cc2c6c2a5d0cb
Sep 29 10:57:03 dist-test-slave-8nrt krb5kdc[22950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679423, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.195@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:57:03.241951 23638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:03.231442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:50293 (local address 127.10.238.254:34825)
0929 10:57:03.231501 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:03.231504 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:03.231782 (+ 278us) server_negotiation.cc:407] Connection header received
0929 10:57:03.232452 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:03.232453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:03.232465 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:03.232524 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:57:03.233240 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:03.234093 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:03.235614 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:03.235916 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:03.235964 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:03.238606 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:57:03.238607 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:57:03.238608 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:57:03.238616 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:57:03.240473 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:03.240912 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:03.240913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:03.240914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:03.240986 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:03.241231 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:03.241232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:03.241232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:03.241408 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:57:03.241480 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:03.241743 (+ 263us) server_negotiation.cc:299] Negotiation successful
0929 10:57:03.241849 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 10:57:03.242060 23912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:03.231144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:34825 (local address 127.10.238.195:50293)
0929 10:57:03.231640 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:03.231688 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:03.232326 (+ 638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:03.232605 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:57:03.232608 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:57:03.232842 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:57:03.233131 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:03.233139 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:03.234211 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:03.234212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:03.235512 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:03.235516 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:03.236068 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:03.236068 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:03.236201 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:03.236877 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:57:03.236887 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:57:03.238504 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:57:03.240570 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:03.240575 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:03.240577 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:03.240824 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:03.241063 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:03.241063 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:03.241063 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:03.241162 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:03.241485 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:57:03.241490 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:57:03.241605 (+ 115us) client_negotiation.cc:769] Sending connection context
0929 10:57:03.241686 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 10:57:03.241836 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":87,"threads_started":1}
I20260929 10:57:03.242516 23910 heartbeater.cc:347] Connected to a master server at 127.10.238.254:34825
I20260929 10:57:03.242616 23910 heartbeater.cc:464] Registering TS with master...
I20260929 10:57:03.242805 23910 heartbeater.cc:511] Master 127.10.238.254:34825 requested a full tablet report, sending...
I20260929 10:57:03.243206 23426 ts_manager.cc:194] Registered new tserver with Master: 1e4c5281b9df4fc58b8cc13dda2ccb61 (127.10.238.195:33633)
I20260929 10:57:03.243665 23426 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.195@KRBTEST.COM'} at 127.10.238.195:50293
I20260929 10:57:03.247321 11195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:57:03.247998 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 23507
I20260929 10:57:03.254577 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 23644
I20260929 10:57:03.260167 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 23780
I20260929 10:57:03.265502 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 23394
I20260929 10:57:03.271898 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:34825
--webserver_interface=127.10.238.254
--webserver_port=35647
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:35359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:34825
--ranger_config_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.238.212:35939/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.238.212:35939/kms with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:57:03.351948 23913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:57:03.352154 23913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:57:03.352174 23913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:57:03.353271 23913 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:57:03.353384 23913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:57:03.353420 23913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:57:03.353439 23913 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:57:03.353458 23913 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:57:03.355059 23913 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:35359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.238.212:35939/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:34825
--ranger_config_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:34825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.238.254
--webserver_port=35647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:57:03.355335 23913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:57:03.355568 23913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:57:03.360369 23919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:03.360347 23918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:03.360357 23921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:57:03.360611 23913 server_base.cc:1034] running on GCE node
I20260929 10:57:03.360844 23913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:57:03.361107 23913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:57:03.362269 23913 hybrid_clock.cc:648] HybridClock initialized: now 1790679423362248 us; error 32 us; skew 500 ppm
Sep 29 10:57:03 dist-test-slave-8nrt krb5kdc[22950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679423, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:57:03.364634 23913 init.cc:377] Logged in from keytab as kudu/127.10.238.254@KRBTEST.COM (short username kudu)
I20260929 10:57:03.365047 23913 webserver.cc:460] Webserver started at http://127.10.238.254:35647/ using document root <none> and password file <none>
I20260929 10:57:03.365321 23913 fs_manager.cc:359] Metadata directory not provided
I20260929 10:57:03.365398 23913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 10:57:03 dist-test-slave-8nrt krb5kdc[22950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679423, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for HTTP/127.10.238.212@KRBTEST.COM
I20260929 10:57:03.421598 23913 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260929 10:57:03.422523 23929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:03.422739 23913 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:57:03.422844 23913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "5011a7bab2194a5985afd5dd97527e21"
format_stamp: "Formatted at 2026-09-29 10:57:01 on dist-test-slave-8nrt"
server_key: "283d7e10f632945e9dc50091bf337940"
server_key_iv: "604101599402b38745cb28a330baa95b"
server_key_version: "kuduclusterkey@0"
I20260929 10:57:03.423323 23913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:57:03.440274 23913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:57:03.441959 23913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:57:03.442124 23913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:57:03.446316 23913 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.254:34825
I20260929 10:57:03.446345 23981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.254:34825 every 8 connection(s)
I20260929 10:57:03.446722 23913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:57:03.447695 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 23913
I20260929 10:57:03.448033 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.238.212:35939/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.193:34175
--local_ip_for_outbound_sockets=127.10.238.193
--tserver_master_addrs=127.10.238.254:34825
--webserver_port=41163
--webserver_interface=127.10.238.193
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab
--principal=kudu/127.10.238.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:35359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.238.212:35939/kms with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:57:03.448112 23982 sys_catalog.cc:263] Verifying existing consensus state
I20260929 10:57:03.449191 23982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5011a7bab2194a5985afd5dd97527e21: Bootstrap starting.
I20260929 10:57:03.457698 23982 log.cc:824] T 00000000000000000000000000000000 P 5011a7bab2194a5985afd5dd97527e21: Log is configured to *not* fsync() on all Append() calls
I20260929 10:57:03.459482 23982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5011a7bab2194a5985afd5dd97527e21: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 10:57:03.459784 23982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5011a7bab2194a5985afd5dd97527e21: Bootstrap complete.
I20260929 10:57:03.462357 23982 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5011a7bab2194a5985afd5dd97527e21 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5011a7bab2194a5985afd5dd97527e21" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 34825 } }
I20260929 10:57:03.462586 23982 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5011a7bab2194a5985afd5dd97527e21 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5011a7bab2194a5985afd5dd97527e21, State: Initialized, Role: FOLLOWER
I20260929 10:57:03.462756 23982 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5011a7bab2194a5985afd5dd97527e21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5011a7bab2194a5985afd5dd97527e21" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 34825 } }
I20260929 10:57:03.462832 23982 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5011a7bab2194a5985afd5dd97527e21 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:57:03.462883 23982 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5011a7bab2194a5985afd5dd97527e21 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:57:03.462930 23982 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5011a7bab2194a5985afd5dd97527e21 [term 1 FOLLOWER]: Advancing to term 2
I20260929 10:57:03.463966 23982 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5011a7bab2194a5985afd5dd97527e21 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5011a7bab2194a5985afd5dd97527e21" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 34825 } }
I20260929 10:57:03.464082 23982 leader_election.cc:302] T 00000000000000000000000000000000 P 5011a7bab2194a5985afd5dd97527e21 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5011a7bab2194a5985afd5dd97527e21; no voters:
I20260929 10:57:03.464280 23982 leader_election.cc:288] T 00000000000000000000000000000000 P 5011a7bab2194a5985afd5dd97527e21 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 10:57:03.464321 23986 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5011a7bab2194a5985afd5dd97527e21 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 10:57:03.464437 23986 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5011a7bab2194a5985afd5dd97527e21 [term 2 LEADER]: Becoming Leader. State: Replica: 5011a7bab2194a5985afd5dd97527e21, State: Running, Role: LEADER
I20260929 10:57:03.464552 23982 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5011a7bab2194a5985afd5dd97527e21 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:57:03.464550 23986 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5011a7bab2194a5985afd5dd97527e21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5011a7bab2194a5985afd5dd97527e21" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 34825 } }
I20260929 10:57:03.464998 23987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5011a7bab2194a5985afd5dd97527e21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5011a7bab2194a5985afd5dd97527e21. Latest consensus state: current_term: 2 leader_uuid: "5011a7bab2194a5985afd5dd97527e21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5011a7bab2194a5985afd5dd97527e21" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 34825 } } }
I20260929 10:57:03.465137 23987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5011a7bab2194a5985afd5dd97527e21 [sys.catalog]: This master's current role is: LEADER
I20260929 10:57:03.465332 23988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5011a7bab2194a5985afd5dd97527e21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5011a7bab2194a5985afd5dd97527e21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5011a7bab2194a5985afd5dd97527e21" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 34825 } } }
I20260929 10:57:03.465440 23988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5011a7bab2194a5985afd5dd97527e21 [sys.catalog]: This master's current role is: LEADER
I20260929 10:57:03.465591 23996 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:57:03.466089 23996 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:57:03.466216 23996 catalog_manager.cc:1259] Loaded cluster ID: 470f7ba3816549969eae461f4a9a4d55
I20260929 10:57:03.466250 23996 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:57:03.466460 23982 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 10:57:03.469705 23996 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:57:03.470358 23996 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5011a7bab2194a5985afd5dd97527e21: Loaded TSK: 0
I20260929 10:57:03.470678 23996 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 10:57:03.557834 23984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:57:03.558091 23984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:57:03.558125 23984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:57:03.559754 23984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:57:03.559815 23984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:57:03.559856 23984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.193
I20260929 10:57:03.561973 23984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:35359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.238.212:35939/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab
--principal=kudu/127.10.238.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.193:34175
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.238.193
--webserver_port=41163
--tserver_master_addrs=127.10.238.254:34825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.193
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:57:03.562211 23984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:57:03.562482 23984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:57:03.568792 24019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:03.568765 24022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:03.569125 24020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:57:03.569373 23984 server_base.cc:1034] running on GCE node
I20260929 10:57:03.569653 23984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:57:03.569912 23984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:57:03.571075 23984 hybrid_clock.cc:648] HybridClock initialized: now 1790679423571046 us; error 43 us; skew 500 ppm
Sep 29 10:57:03 dist-test-slave-8nrt krb5kdc[22950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679423, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:57:03.574594 23984 init.cc:377] Logged in from keytab as kudu/127.10.238.193@KRBTEST.COM (short username kudu)
I20260929 10:57:03.575124 23984 webserver.cc:460] Webserver started at http://127.10.238.193:41163/ using document root <none> and password file <none>
I20260929 10:57:03.575415 23984 fs_manager.cc:359] Metadata directory not provided
I20260929 10:57:03.575492 23984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 10:57:03 dist-test-slave-8nrt krb5kdc[22950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679423, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.193@KRBTEST.COM for HTTP/127.10.238.212@KRBTEST.COM
I20260929 10:57:03.595292 23984 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260929 10:57:03.597831 24030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:03.598014 23984 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 10:57:03.598105 23984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-0/wal
uuid: "133a79f08d534b7f8fea4cac98dfce44"
format_stamp: "Formatted at 2026-09-29 10:57:02 on dist-test-slave-8nrt"
server_key: "0e37fd5a0a2ff8105cf03d57c949a0c0"
server_key_iv: "a955a1d5a29353e18a61ce36075cc15d"
server_key_version: "kuduclusterkey@0"
I20260929 10:57:03.598613 23984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:57:03.619760 23984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:57:03.622156 23984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:57:03.622325 23984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:57:03.622529 23984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:57:03.622875 23984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:57:03.622912 23984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:03.622943 23984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:57:03.622961 23984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:03.630995 23984 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.193:34175
I20260929 10:57:03.631472 23984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
I20260929 10:57:03.632468 24143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.193:34175 every 8 connection(s)
I20260929 10:57:03.636094 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 23984
I20260929 10:57:03.636422 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.238.212:35939/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.194:45691
--local_ip_for_outbound_sockets=127.10.238.194
--tserver_master_addrs=127.10.238.254:34825
--webserver_port=36111
--webserver_interface=127.10.238.194
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab
--principal=kudu/127.10.238.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:35359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.238.212:35939/kms with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 10:57:03 dist-test-slave-8nrt krb5kdc[22950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679423, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.193@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:57:03.654862 24146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:03.632749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:34825 (local address 127.10.238.193:33967)
0929 10:57:03.633029 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:03.633074 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:03.633719 (+ 645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:03.634395 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:57:03.634400 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:57:03.634639 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:57:03.634920 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:03.634926 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:03.636606 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:03.636607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:03.637868 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:03.637871 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:03.638712 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:03.638714 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:03.638871 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:03.639495 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:57:03.639505 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:57:03.641151 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:57:03.652414 (+ 11263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:03.652426 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:03.652429 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:03.652762 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:03.653051 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:03.653052 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:03.653053 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:03.653221 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:03.654061 (+ 840us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:57:03.654069 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:57:03.654248 (+ 179us) client_negotiation.cc:769] Sending connection context
0929 10:57:03.654346 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 10:57:03.654592 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":90,"threads_started":1}
I20260929 10:57:03.656015 24147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:03.632825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:33967 (local address 127.10.238.254:34825)
0929 10:57:03.633931 (+ 1106us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:03.633939 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:03.633972 (+ 33us) server_negotiation.cc:407] Connection header received
0929 10:57:03.634122 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:03.634127 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:03.634180 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:03.634326 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:57:03.635062 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:03.636482 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:03.637968 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:03.638584 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:03.638701 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:03.641313 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:57:03.641317 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:57:03.641318 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:57:03.641330 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:57:03.652257 (+ 10927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:03.652866 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:03.652869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:03.652870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:03.652980 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:03.653371 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:03.653372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:03.653372 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:03.653748 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:57:03.655573 (+ 1825us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:03.655661 (+ 88us) server_negotiation.cc:299] Negotiation successful
0929 10:57:03.655810 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1051,"thread_start_us":85,"threads_started":1}
I20260929 10:57:03.657202 24144 heartbeater.cc:347] Connected to a master server at 127.10.238.254:34825
I20260929 10:57:03.657342 24144 heartbeater.cc:464] Registering TS with master...
I20260929 10:57:03.657524 24144 heartbeater.cc:511] Master 127.10.238.254:34825 requested a full tablet report, sending...
W20260929 10:57:03.658418 24144 heartbeater.cc:644] Failed to heartbeat to 127.10.238.254:34825 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 10:57:03.658494 24144 heartbeater.cc:464] Registering TS with master...
I20260929 10:57:03.658553 24144 heartbeater.cc:511] Master 127.10.238.254:34825 requested a full tablet report, sending...
I20260929 10:57:03.658880 24144 heartbeater.cc:464] Registering TS with master...
I20260929 10:57:03.658954 24144 heartbeater.cc:511] Master 127.10.238.254:34825 requested a full tablet report, sending...
W20260929 10:57:03.659262 24144 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 10:57:03.758916 24148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:57:03.759162 24148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:57:03.759203 24148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:57:03.760428 24148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:57:03.760489 24148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:57:03.760535 24148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.194
I20260929 10:57:03.762342 24148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:35359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.238.212:35939/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab
--principal=kudu/127.10.238.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.194:45691
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.238.194
--webserver_port=36111
--tserver_master_addrs=127.10.238.254:34825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.194
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:57:03.762581 24148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:57:03.762835 24148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:57:03.768850 24153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:03.768860 24156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:03.768994 24154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:57:03.769447 24148 server_base.cc:1034] running on GCE node
I20260929 10:57:03.769649 24148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:57:03.769925 24148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:57:03.771082 24148 hybrid_clock.cc:648] HybridClock initialized: now 1790679423771060 us; error 38 us; skew 500 ppm
Sep 29 10:57:03 dist-test-slave-8nrt krb5kdc[22950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679423, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:57:03.773912 24148 init.cc:377] Logged in from keytab as kudu/127.10.238.194@KRBTEST.COM (short username kudu)
I20260929 10:57:03.774402 24148 webserver.cc:460] Webserver started at http://127.10.238.194:36111/ using document root <none> and password file <none>
I20260929 10:57:03.774636 24148 fs_manager.cc:359] Metadata directory not provided
I20260929 10:57:03.774718 24148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 10:57:03 dist-test-slave-8nrt krb5kdc[22950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679423, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.194@KRBTEST.COM for HTTP/127.10.238.212@KRBTEST.COM
I20260929 10:57:03.791174 24148 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:57:03.793699 24164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:03.793875 24148 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 10:57:03.793959 24148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-1/wal
uuid: "bde3d177c7c84afea69a699e117771f8"
format_stamp: "Formatted at 2026-09-29 10:57:02 on dist-test-slave-8nrt"
server_key: "41048e4567266e5ad99556b655bcd955"
server_key_iv: "9bbb3986ace25ac86dd9e17ba138f39c"
server_key_version: "kuduclusterkey@0"
I20260929 10:57:03.794437 24148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:57:03.822149 24148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:57:03.824594 24148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:57:03.824765 24148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:57:03.824996 24148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:57:03.825397 24148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:57:03.825453 24148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:03.825485 24148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:57:03.825510 24148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:03.832401 24148 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.194:45691
I20260929 10:57:03.832659 24277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.194:45691 every 8 connection(s)
I20260929 10:57:03.832887 24148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
I20260929 10:57:03.842213 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 24148
I20260929 10:57:03.842540 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.238.212:35939/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.195:33633
--local_ip_for_outbound_sockets=127.10.238.195
--tserver_master_addrs=127.10.238.254:34825
--webserver_port=40083
--webserver_interface=127.10.238.195
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab
--principal=kudu/127.10.238.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:35359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.238.212:35939/kms with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 10:57:03 dist-test-slave-8nrt krb5kdc[22950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679423, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.194@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:57:03.852658 24147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:03.833718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:40929 (local address 127.10.238.254:34825)
0929 10:57:03.833978 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:03.833980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:03.841590 (+ 7610us) server_negotiation.cc:407] Connection header received
0929 10:57:03.842364 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:03.842366 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:03.842386 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:03.842419 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:57:03.843311 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:03.844083 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:03.845659 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:03.845946 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:03.846036 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:03.848652 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:57:03.848654 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:57:03.848654 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:57:03.848662 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:57:03.850535 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:03.851152 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:03.851153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:03.851154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:03.851221 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:03.851550 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:03.851551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:03.851551 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:03.851696 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:57:03.851745 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:03.852460 (+ 715us) server_negotiation.cc:299] Negotiation successful
0929 10:57:03.852557 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228}
I20260929 10:57:03.854502 24280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:03.833723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:34825 (local address 127.10.238.194:40929)
0929 10:57:03.841409 (+ 7686us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:03.841465 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:03.842137 (+ 672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:03.842625 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:57:03.842630 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:57:03.842876 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:57:03.843171 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:03.843178 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:03.844211 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:03.844212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:03.845556 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:03.845559 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:03.846081 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:03.846082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:03.846223 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:03.846863 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:57:03.846874 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:57:03.848541 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:57:03.850718 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:03.850725 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:03.850727 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:03.851041 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:03.851316 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:03.851317 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:03.851317 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:03.851463 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:03.851799 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:57:03.851805 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:57:03.852265 (+ 460us) client_negotiation.cc:769] Sending connection context
0929 10:57:03.854059 (+ 1794us) client_negotiation.cc:240] Negotiation successful
0929 10:57:03.854253 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7623,"thread_start_us":87,"threads_started":1}
I20260929 10:57:03.855199 24278 heartbeater.cc:347] Connected to a master server at 127.10.238.254:34825
I20260929 10:57:03.855314 24278 heartbeater.cc:464] Registering TS with master...
I20260929 10:57:03.855512 24278 heartbeater.cc:511] Master 127.10.238.254:34825 requested a full tablet report, sending...
W20260929 10:57:03.856249 24278 heartbeater.cc:644] Failed to heartbeat to 127.10.238.254:34825 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 10:57:03.856325 24278 heartbeater.cc:464] Registering TS with master...
I20260929 10:57:03.856407 24278 heartbeater.cc:511] Master 127.10.238.254:34825 requested a full tablet report, sending...
I20260929 10:57:03.856761 24278 heartbeater.cc:464] Registering TS with master...
I20260929 10:57:03.856834 24278 heartbeater.cc:511] Master 127.10.238.254:34825 requested a full tablet report, sending...
W20260929 10:57:03.857173 24278 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 10:57:03.969573 24281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:57:03.969849 24281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:57:03.969884 24281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:57:03.971534 24281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:57:03.971585 24281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:57:03.971626 24281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.195
I20260929 10:57:03.973860 24281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:35359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.238.212:35939/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab
--principal=kudu/127.10.238.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.195:33633
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.238.195
--webserver_port=40083
--tserver_master_addrs=127.10.238.254:34825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.195
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:57:03.974128 24281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:57:03.974385 24281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:57:03.980896 24290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:03.980892 24288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:03.981658 24287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:57:03.981886 24281 server_base.cc:1034] running on GCE node
I20260929 10:57:03.982192 24281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:57:03.982462 24281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:57:03.997884 24281 hybrid_clock.cc:648] HybridClock initialized: now 1790679423997851 us; error 53 us; skew 500 ppm
Sep 29 10:57:03 dist-test-slave-8nrt krb5kdc[22950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679423, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:57:04.001248 24281 init.cc:377] Logged in from keytab as kudu/127.10.238.195@KRBTEST.COM (short username kudu)
I20260929 10:57:04.001813 24281 webserver.cc:460] Webserver started at http://127.10.238.195:40083/ using document root <none> and password file <none>
I20260929 10:57:04.002331 24281 fs_manager.cc:359] Metadata directory not provided
I20260929 10:57:04.002463 24281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 10:57:04 dist-test-slave-8nrt krb5kdc[22950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679423, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.195@KRBTEST.COM for HTTP/127.10.238.212@KRBTEST.COM
I20260929 10:57:04.022082 24281 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.001s
I20260929 10:57:04.022841 24298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:04.023020 24281 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 10:57:04.023094 24281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-2/wal
uuid: "1e4c5281b9df4fc58b8cc13dda2ccb61"
format_stamp: "Formatted at 2026-09-29 10:57:03 on dist-test-slave-8nrt"
server_key: "96604afa1d6a9fcfcc76e8ece88ffae1"
server_key_iv: "57a7606968a669491241618ef39de5e7"
server_key_version: "kuduclusterkey@0"
I20260929 10:57:04.023545 24281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:57:04.058400 24281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:57:04.061815 24281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:57:04.061990 24281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:57:04.062225 24281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:57:04.062588 24281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:57:04.062628 24281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:04.062659 24281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:57:04.062676 24281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:04.069823 24281 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.195:33633
I20260929 10:57:04.070279 24281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
I20260929 10:57:04.075788 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 24281
I20260929 10:57:04.077654 24411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.195:33633 every 8 connection(s)
Sep 29 10:57:04 dist-test-slave-8nrt krb5kdc[22950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679423, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.195@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:57:04.087095 24147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:04.071100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:51881 (local address 127.10.238.254:34825)
0929 10:57:04.071154 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:04.071156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:04.077559 (+ 6403us) server_negotiation.cc:407] Connection header received
0929 10:57:04.078223 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:04.078225 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:04.078249 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:04.078306 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:57:04.078917 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:04.079421 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:04.080476 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:04.080681 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:04.080761 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:04.083553 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:57:04.083554 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:57:04.083554 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:57:04.083565 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:57:04.085428 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:04.085941 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:04.085942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:04.085942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:04.086002 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:04.086321 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:04.086322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:04.086323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:04.086459 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:57:04.086504 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:04.086900 (+ 396us) server_negotiation.cc:299] Negotiation successful
0929 10:57:04.086992 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 10:57:04.087684 24414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:04.071114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:34825 (local address 127.10.238.195:51881)
0929 10:57:04.077406 (+ 6292us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:04.077474 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:04.078108 (+ 634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:04.078383 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:57:04.078388 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:57:04.078595 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:57:04.078814 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:04.078819 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:04.079537 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:04.079539 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:04.080380 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:04.080382 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:04.080787 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:04.080789 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:04.080919 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:04.081572 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:57:04.081583 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:57:04.083453 (+ 1870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:57:04.085521 (+ 2068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:04.085529 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:04.085531 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:04.085840 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:04.086077 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:04.086078 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:04.086078 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:04.086238 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:04.086555 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:57:04.086561 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:57:04.086730 (+ 169us) client_negotiation.cc:769] Sending connection context
0929 10:57:04.087241 (+ 511us) client_negotiation.cc:240] Negotiation successful
0929 10:57:04.087411 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6233,"thread_start_us":80,"threads_started":1}
I20260929 10:57:04.088136 24412 heartbeater.cc:347] Connected to a master server at 127.10.238.254:34825
I20260929 10:57:04.088241 24412 heartbeater.cc:464] Registering TS with master...
I20260929 10:57:04.088426 24412 heartbeater.cc:511] Master 127.10.238.254:34825 requested a full tablet report, sending...
W20260929 10:57:04.089145 24412 heartbeater.cc:644] Failed to heartbeat to 127.10.238.254:34825 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 10:57:04.089227 24412 heartbeater.cc:464] Registering TS with master...
I20260929 10:57:04.089331 24412 heartbeater.cc:511] Master 127.10.238.254:34825 requested a full tablet report, sending...
I20260929 10:57:04.089612 24412 heartbeater.cc:464] Registering TS with master...
I20260929 10:57:04.089680 24412 heartbeater.cc:511] Master 127.10.238.254:34825 requested a full tablet report, sending...
W20260929 10:57:04.089956 24412 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 10:57:04.099638 24416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:04.076246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33792 (local address 127.10.238.254:34825)
0929 10:57:04.081575 (+ 5329us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:04.081577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:04.081587 (+ 10us) server_negotiation.cc:407] Connection header received
0929 10:57:04.081611 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:04.081611 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:04.081623 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:04.081707 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:57:04.093309 (+ 11602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:04.094064 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:04.095029 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:04.095328 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:04.095405 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:04.096720 (+ 1315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:57:04.096722 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:57:04.096722 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:57:04.096730 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:57:04.098352 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:04.098788 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:04.098789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:04.098790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:04.098885 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:04.099077 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:04.099078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:04.099078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:04.099189 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:57:04.099386 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:04.099426 (+ 40us) server_negotiation.cc:299] Negotiation successful
0929 10:57:04.099533 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5305,"thread_start_us":84,"threads_started":1}
Sep 29 10:57:04 dist-test-slave-8nrt krb5kdc[22950](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790679424, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:57:04.659482 24144 heartbeater.cc:464] Registering TS with master...
I20260929 10:57:04.659642 24144 heartbeater.cc:511] Master 127.10.238.254:34825 requested a full tablet report, sending...
I20260929 10:57:04.660079 23946 ts_manager.cc:194] Registered new tserver with Master: 133a79f08d534b7f8fea4cac98dfce44 (127.10.238.193:34175)
I20260929 10:57:04.661302 23946 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.193@KRBTEST.COM'} at 127.10.238.193:33967
I20260929 10:57:04.857321 24278 heartbeater.cc:464] Registering TS with master...
I20260929 10:57:04.857491 24278 heartbeater.cc:511] Master 127.10.238.254:34825 requested a full tablet report, sending...
I20260929 10:57:04.858053 23946 ts_manager.cc:194] Registered new tserver with Master: bde3d177c7c84afea69a699e117771f8 (127.10.238.194:45691)
I20260929 10:57:04.858558 23946 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.194@KRBTEST.COM'} at 127.10.238.194:40929
Sep 29 10:57:04 dist-test-slave-8nrt krb5kdc[22950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679424, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for HTTP/127.10.238.212@KRBTEST.COM
I20260929 10:57:05.090126 24412 heartbeater.cc:464] Registering TS with master...
I20260929 10:57:05.090309 24412 heartbeater.cc:511] Master 127.10.238.254:34825 requested a full tablet report, sending...
I20260929 10:57:05.090721 23946 ts_manager.cc:194] Registered new tserver with Master: 1e4c5281b9df4fc58b8cc13dda2ccb61 (127.10.238.195:33633)
I20260929 10:57:05.091269 23946 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.195@KRBTEST.COM'} at 127.10.238.195:51881
I20260929 10:57:05.091953 11195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:57:05.101776 24434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:05.092984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33808 (local address 127.10.238.254:34825)
0929 10:57:05.093124 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:05.093126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:05.093136 (+ 10us) server_negotiation.cc:407] Connection header received
0929 10:57:05.093206 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:05.093207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:05.093219 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:05.093322 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:57:05.094027 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:05.094866 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:05.095927 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:05.096273 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:05.096357 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:05.097931 (+ 1574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:57:05.097933 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:57:05.097933 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:57:05.097942 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:57:05.100431 (+ 2489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:05.100842 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:05.100844 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:05.100844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:05.100918 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:05.101150 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:05.101152 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:05.101152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:05.101331 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:57:05.101438 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:05.101555 (+ 117us) server_negotiation.cc:299] Negotiation successful
0929 10:57:05.101660 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":81,"threads_started":1}
I20260929 10:57:05.104007 23946 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33808:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:57:05.106545 23946 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:57:05.117857 24444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:05.109673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:34175 (local address 127.0.0.1:52424)
0929 10:57:05.110740 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:05.110781 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:05.110844 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:05.113875 (+ 3031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:57:05.113876 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:57:05.113878 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:57:05.113924 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:05.113925 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:05.115188 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:05.115189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:05.116488 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:05.116490 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:05.117564 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:05.117565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:05.117647 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:05.117660 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 10:57:05.117724 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 10:57:05.117774 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1039,"thread_start_us":85,"threads_started":1}
I20260929 10:57:05.118422 24442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:05.109931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52424 (local address 127.10.238.193:34175)
0929 10:57:05.113654 (+ 3723us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:05.113659 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:05.113668 (+ 9us) server_negotiation.cc:407] Connection header received
0929 10:57:05.113702 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:05.113704 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:05.113719 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:05.113788 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:57:05.114000 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:05.115100 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:05.116587 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:05.117404 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:05.117476 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:05.117531 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:05.118285 (+ 754us) server_negotiation.cc:299] Negotiation successful
0929 10:57:05.118329 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3685,"thread_start_us":61,"threads_started":1}
I20260929 10:57:05.119210 24064 tablet_service.cc:1467] Processing CreateTablet for tablet cd4a4760cabd45deadab9a134668f8d1 (DEFAULT_TABLE table=test-table [id=d99e97fa33834400a48efdd772db8776]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:57:05.119431 24440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:05.110140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.195:33633 (local address 127.0.0.1:36400)
0929 10:57:05.113507 (+ 3367us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:05.113515 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:05.113551 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:05.114093 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:57:05.114093 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:57:05.114096 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:57:05.114130 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:05.114131 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:05.116827 (+ 2696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:05.116828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:05.118251 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:05.118254 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:05.119214 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:05.119215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:05.119278 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:05.119284 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 10:57:05.119316 (+ 32us) client_negotiation.cc:240] Negotiation successful
0929 10:57:05.119356 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3356,"thread_start_us":65,"threads_started":1}
I20260929 10:57:05.119625 24064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd4a4760cabd45deadab9a134668f8d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:57:05.119805 24445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:05.110275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36400 (local address 127.10.238.195:33633)
0929 10:57:05.110986 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:05.110991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:05.113897 (+ 2906us) server_negotiation.cc:407] Connection header received
0929 10:57:05.113936 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:05.113938 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:05.113954 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:05.114024 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:57:05.114202 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:05.115533 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:05.118354 (+ 2821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:05.119105 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:05.119597 (+ 492us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:05.119670 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:05.119695 (+ 25us) server_negotiation.cc:299] Negotiation successful
0929 10:57:05.119734 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":69,"threads_started":1}
I20260929 10:57:05.120381 24335 tablet_service.cc:1467] Processing CreateTablet for tablet cd4a4760cabd45deadab9a134668f8d1 (DEFAULT_TABLE table=test-table [id=d99e97fa33834400a48efdd772db8776]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:57:05.120721 24335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd4a4760cabd45deadab9a134668f8d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:57:05.126966 24441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:05.109719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.194:45691 (local address 127.0.0.1:39258)
0929 10:57:05.113361 (+ 3642us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:05.113373 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:05.113434 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:05.113734 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:57:05.113737 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:57:05.113740 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:57:05.113819 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:05.113820 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:05.124443 (+ 10623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:05.124444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:05.125660 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:05.125664 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:05.126677 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:05.126678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:05.126780 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:05.126785 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 10:57:05.126835 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 10:57:05.126884 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3618,"thread_start_us":91,"threads_started":1}
I20260929 10:57:05.127013 24443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:05.109941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39258 (local address 127.10.238.194:45691)
0929 10:57:05.111054 (+ 1113us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:05.111057 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:05.113452 (+ 2395us) server_negotiation.cc:407] Connection header received
0929 10:57:05.113498 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:05.113502 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:05.113519 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:05.113607 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:57:05.115627 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:05.116713 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:05.125773 (+ 9060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:05.126573 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:05.126666 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:05.126723 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:05.126850 (+ 127us) server_negotiation.cc:299] Negotiation successful
0929 10:57:05.126933 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1085,"thread_start_us":54,"threads_started":1}
I20260929 10:57:05.128091 24212 tablet_service.cc:1467] Processing CreateTablet for tablet cd4a4760cabd45deadab9a134668f8d1 (DEFAULT_TABLE table=test-table [id=d99e97fa33834400a48efdd772db8776]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:57:05.128464 24212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd4a4760cabd45deadab9a134668f8d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:57:05.132014 24448 tablet_bootstrap.cc:492] T cd4a4760cabd45deadab9a134668f8d1 P bde3d177c7c84afea69a699e117771f8: Bootstrap starting.
I20260929 10:57:05.132669 24448 tablet_bootstrap.cc:654] T cd4a4760cabd45deadab9a134668f8d1 P bde3d177c7c84afea69a699e117771f8: Neither blocks nor log segments found. Creating new log.
I20260929 10:57:05.133041 24448 log.cc:824] T cd4a4760cabd45deadab9a134668f8d1 P bde3d177c7c84afea69a699e117771f8: Log is configured to *not* fsync() on all Append() calls
I20260929 10:57:05.133883 24448 tablet_bootstrap.cc:492] T cd4a4760cabd45deadab9a134668f8d1 P bde3d177c7c84afea69a699e117771f8: No bootstrap required, opened a new log
I20260929 10:57:05.133957 24448 ts_tablet_manager.cc:1397] T cd4a4760cabd45deadab9a134668f8d1 P bde3d177c7c84afea69a699e117771f8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260929 10:57:05.136404 24448 raft_consensus.cc:348] T cd4a4760cabd45deadab9a134668f8d1 P bde3d177c7c84afea69a699e117771f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133a79f08d534b7f8fea4cac98dfce44" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34175 } } peers { permanent_uuid: "bde3d177c7c84afea69a699e117771f8" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 45691 } } peers { permanent_uuid: "1e4c5281b9df4fc58b8cc13dda2ccb61" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 33633 } }
I20260929 10:57:05.136564 24448 raft_consensus.cc:374] T cd4a4760cabd45deadab9a134668f8d1 P bde3d177c7c84afea69a699e117771f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:57:05.136595 24448 raft_consensus.cc:729] T cd4a4760cabd45deadab9a134668f8d1 P bde3d177c7c84afea69a699e117771f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bde3d177c7c84afea69a699e117771f8, State: Initialized, Role: FOLLOWER
I20260929 10:57:05.136715 24448 consensus_queue.cc:260] T cd4a4760cabd45deadab9a134668f8d1 P bde3d177c7c84afea69a699e117771f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133a79f08d534b7f8fea4cac98dfce44" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34175 } } peers { permanent_uuid: "bde3d177c7c84afea69a699e117771f8" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 45691 } } peers { permanent_uuid: "1e4c5281b9df4fc58b8cc13dda2ccb61" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 33633 } }
I20260929 10:57:05.137029 24278 heartbeater.cc:503] Master 127.10.238.254:34825 was elected leader, sending a full tablet report...
I20260929 10:57:05.137410 24448 ts_tablet_manager.cc:1428] T cd4a4760cabd45deadab9a134668f8d1 P bde3d177c7c84afea69a699e117771f8: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260929 10:57:05.137524 24447 tablet_bootstrap.cc:492] T cd4a4760cabd45deadab9a134668f8d1 P 1e4c5281b9df4fc58b8cc13dda2ccb61: Bootstrap starting.
I20260929 10:57:05.138267 24447 tablet_bootstrap.cc:654] T cd4a4760cabd45deadab9a134668f8d1 P 1e4c5281b9df4fc58b8cc13dda2ccb61: Neither blocks nor log segments found. Creating new log.
I20260929 10:57:05.138540 24447 log.cc:824] T cd4a4760cabd45deadab9a134668f8d1 P 1e4c5281b9df4fc58b8cc13dda2ccb61: Log is configured to *not* fsync() on all Append() calls
I20260929 10:57:05.138871 24446 tablet_bootstrap.cc:492] T cd4a4760cabd45deadab9a134668f8d1 P 133a79f08d534b7f8fea4cac98dfce44: Bootstrap starting.
I20260929 10:57:05.139398 24446 tablet_bootstrap.cc:654] T cd4a4760cabd45deadab9a134668f8d1 P 133a79f08d534b7f8fea4cac98dfce44: Neither blocks nor log segments found. Creating new log.
I20260929 10:57:05.139621 24446 log.cc:824] T cd4a4760cabd45deadab9a134668f8d1 P 133a79f08d534b7f8fea4cac98dfce44: Log is configured to *not* fsync() on all Append() calls
I20260929 10:57:05.140144 24447 tablet_bootstrap.cc:492] T cd4a4760cabd45deadab9a134668f8d1 P 1e4c5281b9df4fc58b8cc13dda2ccb61: No bootstrap required, opened a new log
I20260929 10:57:05.140208 24447 ts_tablet_manager.cc:1397] T cd4a4760cabd45deadab9a134668f8d1 P 1e4c5281b9df4fc58b8cc13dda2ccb61: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 10:57:05.141671 24446 tablet_bootstrap.cc:492] T cd4a4760cabd45deadab9a134668f8d1 P 133a79f08d534b7f8fea4cac98dfce44: No bootstrap required, opened a new log
I20260929 10:57:05.141744 24446 ts_tablet_manager.cc:1397] T cd4a4760cabd45deadab9a134668f8d1 P 133a79f08d534b7f8fea4cac98dfce44: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 10:57:05.142285 24447 raft_consensus.cc:348] T cd4a4760cabd45deadab9a134668f8d1 P 1e4c5281b9df4fc58b8cc13dda2ccb61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133a79f08d534b7f8fea4cac98dfce44" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34175 } } peers { permanent_uuid: "bde3d177c7c84afea69a699e117771f8" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 45691 } } peers { permanent_uuid: "1e4c5281b9df4fc58b8cc13dda2ccb61" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 33633 } }
I20260929 10:57:05.142493 24447 raft_consensus.cc:374] T cd4a4760cabd45deadab9a134668f8d1 P 1e4c5281b9df4fc58b8cc13dda2ccb61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:57:05.142532 24447 raft_consensus.cc:729] T cd4a4760cabd45deadab9a134668f8d1 P 1e4c5281b9df4fc58b8cc13dda2ccb61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e4c5281b9df4fc58b8cc13dda2ccb61, State: Initialized, Role: FOLLOWER
I20260929 10:57:05.142709 24447 consensus_queue.cc:260] T cd4a4760cabd45deadab9a134668f8d1 P 1e4c5281b9df4fc58b8cc13dda2ccb61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133a79f08d534b7f8fea4cac98dfce44" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34175 } } peers { permanent_uuid: "bde3d177c7c84afea69a699e117771f8" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 45691 } } peers { permanent_uuid: "1e4c5281b9df4fc58b8cc13dda2ccb61" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 33633 } }
I20260929 10:57:05.143013 24447 ts_tablet_manager.cc:1428] T cd4a4760cabd45deadab9a134668f8d1 P 1e4c5281b9df4fc58b8cc13dda2ccb61: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 10:57:05.143061 24412 heartbeater.cc:503] Master 127.10.238.254:34825 was elected leader, sending a full tablet report...
I20260929 10:57:05.144455 24446 raft_consensus.cc:348] T cd4a4760cabd45deadab9a134668f8d1 P 133a79f08d534b7f8fea4cac98dfce44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133a79f08d534b7f8fea4cac98dfce44" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34175 } } peers { permanent_uuid: "bde3d177c7c84afea69a699e117771f8" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 45691 } } peers { permanent_uuid: "1e4c5281b9df4fc58b8cc13dda2ccb61" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 33633 } }
I20260929 10:57:05.144627 24446 raft_consensus.cc:374] T cd4a4760cabd45deadab9a134668f8d1 P 133a79f08d534b7f8fea4cac98dfce44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:57:05.144760 24446 raft_consensus.cc:729] T cd4a4760cabd45deadab9a134668f8d1 P 133a79f08d534b7f8fea4cac98dfce44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 133a79f08d534b7f8fea4cac98dfce44, State: Initialized, Role: FOLLOWER
I20260929 10:57:05.145000 24446 consensus_queue.cc:260] T cd4a4760cabd45deadab9a134668f8d1 P 133a79f08d534b7f8fea4cac98dfce44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133a79f08d534b7f8fea4cac98dfce44" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34175 } } peers { permanent_uuid: "bde3d177c7c84afea69a699e117771f8" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 45691 } } peers { permanent_uuid: "1e4c5281b9df4fc58b8cc13dda2ccb61" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 33633 } }
I20260929 10:57:05.145360 24446 ts_tablet_manager.cc:1428] T cd4a4760cabd45deadab9a134668f8d1 P 133a79f08d534b7f8fea4cac98dfce44: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260929 10:57:05.145483 24144 heartbeater.cc:503] Master 127.10.238.254:34825 was elected leader, sending a full tablet report...
Sep 29 10:57:05 dist-test-slave-8nrt krb5kdc[22950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679424, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for HTTP/127.10.238.212@KRBTEST.COM
W20260929 10:57:05.328720 24413 tablet.cc:2369] T cd4a4760cabd45deadab9a134668f8d1 P 1e4c5281b9df4fc58b8cc13dda2ccb61: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 10:57:05.334482 24279 tablet.cc:2369] T cd4a4760cabd45deadab9a134668f8d1 P bde3d177c7c84afea69a699e117771f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:57:05.336633 24454 raft_consensus.cc:482] T cd4a4760cabd45deadab9a134668f8d1 P 133a79f08d534b7f8fea4cac98dfce44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:57:05.336755 24454 raft_consensus.cc:504] T cd4a4760cabd45deadab9a134668f8d1 P 133a79f08d534b7f8fea4cac98dfce44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133a79f08d534b7f8fea4cac98dfce44" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34175 } } peers { permanent_uuid: "bde3d177c7c84afea69a699e117771f8" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 45691 } } peers { permanent_uuid: "1e4c5281b9df4fc58b8cc13dda2ccb61" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 33633 } }
I20260929 10:57:05.337131 24454 leader_election.cc:288] T cd4a4760cabd45deadab9a134668f8d1 P 133a79f08d534b7f8fea4cac98dfce44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bde3d177c7c84afea69a699e117771f8 (127.10.238.194:45691), 1e4c5281b9df4fc58b8cc13dda2ccb61 (127.10.238.195:33633)
I20260929 10:57:05.348891 24445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:05.337750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:49027 (local address 127.10.238.195:33633)
0929 10:57:05.337805 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:05.337807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:05.345503 (+ 7696us) server_negotiation.cc:407] Connection header received
0929 10:57:05.345619 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:05.345621 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:05.345632 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:05.345664 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:57:05.345916 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:05.346722 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:05.347696 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:05.348294 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:05.348372 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:05.348382 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:05.348758 (+ 376us) server_negotiation.cc:299] Negotiation successful
0929 10:57:05.348800 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 10:57:05.349396 24457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:05.337417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.195:33633 (local address 127.10.238.193:49027)
0929 10:57:05.345410 (+ 7993us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:05.345426 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:05.345539 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:05.345741 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:57:05.345742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:57:05.345744 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:57:05.345828 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:05.345830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:05.346811 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:05.346811 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:05.347597 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:05.347599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:05.348595 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:05.348596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:05.348674 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:05.348678 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 10:57:05.349224 (+ 546us) client_negotiation.cc:240] Negotiation successful
0929 10:57:05.349296 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7961,"thread_start_us":107,"threads_started":1}
I20260929 10:57:05.349886 24349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd4a4760cabd45deadab9a134668f8d1" candidate_uuid: "133a79f08d534b7f8fea4cac98dfce44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e4c5281b9df4fc58b8cc13dda2ccb61" is_pre_election: true
I20260929 10:57:05.350086 24349 raft_consensus.cc:2454] T cd4a4760cabd45deadab9a134668f8d1 P 1e4c5281b9df4fc58b8cc13dda2ccb61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 133a79f08d534b7f8fea4cac98dfce44 in term 0.
I20260929 10:57:05.350370 24031 leader_election.cc:302] T cd4a4760cabd45deadab9a134668f8d1 P 133a79f08d534b7f8fea4cac98dfce44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 133a79f08d534b7f8fea4cac98dfce44, 1e4c5281b9df4fc58b8cc13dda2ccb61; no voters:
I20260929 10:57:05.350490 24454 raft_consensus.cc:2781] T cd4a4760cabd45deadab9a134668f8d1 P 133a79f08d534b7f8fea4cac98dfce44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:57:05.350544 24454 raft_consensus.cc:482] T cd4a4760cabd45deadab9a134668f8d1 P 133a79f08d534b7f8fea4cac98dfce44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:57:05.350570 24454 raft_consensus.cc:3037] T cd4a4760cabd45deadab9a134668f8d1 P 133a79f08d534b7f8fea4cac98dfce44 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:57:05.351280 24454 raft_consensus.cc:504] T cd4a4760cabd45deadab9a134668f8d1 P 133a79f08d534b7f8fea4cac98dfce44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133a79f08d534b7f8fea4cac98dfce44" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34175 } } peers { permanent_uuid: "bde3d177c7c84afea69a699e117771f8" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 45691 } } peers { permanent_uuid: "1e4c5281b9df4fc58b8cc13dda2ccb61" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 33633 } }
I20260929 10:57:05.351431 24454 leader_election.cc:288] T cd4a4760cabd45deadab9a134668f8d1 P 133a79f08d534b7f8fea4cac98dfce44 [CANDIDATE]: Term 1 election: Requested vote from peers bde3d177c7c84afea69a699e117771f8 (127.10.238.194:45691), 1e4c5281b9df4fc58b8cc13dda2ccb61 (127.10.238.195:33633)
I20260929 10:57:05.351684 24349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd4a4760cabd45deadab9a134668f8d1" candidate_uuid: "133a79f08d534b7f8fea4cac98dfce44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e4c5281b9df4fc58b8cc13dda2ccb61"
I20260929 10:57:05.351763 24349 raft_consensus.cc:3037] T cd4a4760cabd45deadab9a134668f8d1 P 1e4c5281b9df4fc58b8cc13dda2ccb61 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:57:05.352572 24349 raft_consensus.cc:2454] T cd4a4760cabd45deadab9a134668f8d1 P 1e4c5281b9df4fc58b8cc13dda2ccb61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 133a79f08d534b7f8fea4cac98dfce44 in term 1.
I20260929 10:57:05.352741 24031 leader_election.cc:302] T cd4a4760cabd45deadab9a134668f8d1 P 133a79f08d534b7f8fea4cac98dfce44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 133a79f08d534b7f8fea4cac98dfce44, 1e4c5281b9df4fc58b8cc13dda2ccb61; no voters:
I20260929 10:57:05.353402 24454 raft_consensus.cc:2781] T cd4a4760cabd45deadab9a134668f8d1 P 133a79f08d534b7f8fea4cac98dfce44 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:57:05.353615 24454 raft_consensus.cc:686] T cd4a4760cabd45deadab9a134668f8d1 P 133a79f08d534b7f8fea4cac98dfce44 [term 1 LEADER]: Becoming Leader. State: Replica: 133a79f08d534b7f8fea4cac98dfce44, State: Running, Role: LEADER
I20260929 10:57:05.353719 24454 consensus_queue.cc:237] T cd4a4760cabd45deadab9a134668f8d1 P 133a79f08d534b7f8fea4cac98dfce44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133a79f08d534b7f8fea4cac98dfce44" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34175 } } peers { permanent_uuid: "bde3d177c7c84afea69a699e117771f8" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 45691 } } peers { permanent_uuid: "1e4c5281b9df4fc58b8cc13dda2ccb61" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 33633 } }
I20260929 10:57:05.354628 23946 catalog_manager.cc:5495] T cd4a4760cabd45deadab9a134668f8d1 P 133a79f08d534b7f8fea4cac98dfce44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 133a79f08d534b7f8fea4cac98dfce44 (127.10.238.193). New cstate: current_term: 1 leader_uuid: "133a79f08d534b7f8fea4cac98dfce44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133a79f08d534b7f8fea4cac98dfce44" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 34175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bde3d177c7c84afea69a699e117771f8" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 45691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e4c5281b9df4fc58b8cc13dda2ccb61" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 33633 } health_report { overall_health: UNKNOWN } } }
I20260929 10:57:05.355332 24443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:05.337990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:55963 (local address 127.10.238.194:45691)
0929 10:57:05.338034 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:05.338036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:05.348406 (+ 10370us) server_negotiation.cc:407] Connection header received
0929 10:57:05.348437 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:05.348439 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:05.348449 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:05.348512 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:57:05.348709 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:05.349412 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:05.352856 (+ 3444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:05.353306 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:05.353359 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:05.353367 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:05.355221 (+ 1854us) server_negotiation.cc:299] Negotiation successful
0929 10:57:05.355270 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 10:57:05.355315 24458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:05.337821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.194:45691 (local address 127.10.238.193:55963)
0929 10:57:05.347752 (+ 9931us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:05.347761 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:05.347807 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:05.348529 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:57:05.348530 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:57:05.348532 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:57:05.348628 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:05.348629 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:05.349515 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:05.349516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:05.352545 (+ 3029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:05.352548 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:05.355069 (+ 2521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:05.355070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:05.355149 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:05.355154 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 10:57:05.355201 (+ 47us) client_negotiation.cc:240] Negotiation successful
0929 10:57:05.355242 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9911,"thread_start_us":74,"threads_started":1}
I20260929 10:57:05.356068 24222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd4a4760cabd45deadab9a134668f8d1" candidate_uuid: "133a79f08d534b7f8fea4cac98dfce44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bde3d177c7c84afea69a699e117771f8"
I20260929 10:57:05.356195 24222 raft_consensus.cc:3037] T cd4a4760cabd45deadab9a134668f8d1 P bde3d177c7c84afea69a699e117771f8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:57:05.357082 24222 raft_consensus.cc:2454] T cd4a4760cabd45deadab9a134668f8d1 P bde3d177c7c84afea69a699e117771f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 133a79f08d534b7f8fea4cac98dfce44 in term 1.
I20260929 10:57:05.356065 24221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd4a4760cabd45deadab9a134668f8d1" candidate_uuid: "133a79f08d534b7f8fea4cac98dfce44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bde3d177c7c84afea69a699e117771f8" is_pre_election: true
I20260929 10:57:05.357427 24221 raft_consensus.cc:2362] T cd4a4760cabd45deadab9a134668f8d1 P bde3d177c7c84afea69a699e117771f8 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 133a79f08d534b7f8fea4cac98dfce44 in term 1. Re-sending same reply.
W20260929 10:57:05.383010 24145 tablet.cc:2369] T cd4a4760cabd45deadab9a134668f8d1 P 133a79f08d534b7f8fea4cac98dfce44: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:57:05.400120 24442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:05.396409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33876 (local address 127.10.238.193:34175)
0929 10:57:05.396507 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:05.396509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:05.396516 (+ 7us) server_negotiation.cc:407] Connection header received
0929 10:57:05.396557 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:05.396558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:05.396578 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:05.396728 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:57:05.396935 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:05.397706 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:05.398913 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:05.399217 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:05.399279 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:05.399520 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:57:05.399657 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:57:05.399740 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:05.400014 (+ 274us) server_negotiation.cc:299] Negotiation successful
0929 10:57:05.400037 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 10:57:05.402581 24349 raft_consensus.cc:1261] T cd4a4760cabd45deadab9a134668f8d1 P 1e4c5281b9df4fc58b8cc13dda2ccb61 [term 1 FOLLOWER]: Refusing update from remote peer 133a79f08d534b7f8fea4cac98dfce44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:57:05.402971 24454 consensus_queue.cc:1035] T cd4a4760cabd45deadab9a134668f8d1 P 133a79f08d534b7f8fea4cac98dfce44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e4c5281b9df4fc58b8cc13dda2ccb61" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 33633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:57:05.405246 24221 raft_consensus.cc:1261] T cd4a4760cabd45deadab9a134668f8d1 P bde3d177c7c84afea69a699e117771f8 [term 1 FOLLOWER]: Refusing update from remote peer 133a79f08d534b7f8fea4cac98dfce44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:57:05.405543 24454 consensus_queue.cc:1035] T cd4a4760cabd45deadab9a134668f8d1 P 133a79f08d534b7f8fea4cac98dfce44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bde3d177c7c84afea69a699e117771f8" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:57:05.409921 24462 mvcc.cc:205] Tried to move back new op lower bound from 7334622926445477888 to 7334622926249246720. Current Snapshot: MvccSnapshot[applied={T|T < 7334622926445477888}]
I20260929 10:57:05.410890 24465 mvcc.cc:205] Tried to move back new op lower bound from 7334622926445477888 to 7334622926249246720. Current Snapshot: MvccSnapshot[applied={T|T < 7334622926445477888}]
I20260929 10:57:05.414256 23946 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:57:05.414359 23946 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:57:05.415351 23946 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5011a7bab2194a5985afd5dd97527e21: Sending DeleteTablet for 3 replicas of tablet cd4a4760cabd45deadab9a134668f8d1
I20260929 10:57:05.415659 24212 tablet_service.cc:1514] Processing DeleteTablet for tablet cd4a4760cabd45deadab9a134668f8d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:57:05 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:39258
I20260929 10:57:05.415859 24064 tablet_service.cc:1514] Processing DeleteTablet for tablet cd4a4760cabd45deadab9a134668f8d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:57:05 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:52424
I20260929 10:57:05.416005 24463 mvcc.cc:205] Tried to move back new op lower bound from 7334622926445477888 to 7334622926249246720. Current Snapshot: MvccSnapshot[applied={T|T < 7334622926445477888}]
I20260929 10:57:05.416047 24078 tablet_service.cc:1514] Processing DeleteTablet for tablet cd4a4760cabd45deadab9a134668f8d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:57:05 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:52424
I20260929 10:57:05.416170 24335 tablet_service.cc:1514] Processing DeleteTablet for tablet cd4a4760cabd45deadab9a134668f8d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:57:05 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:36400
I20260929 10:57:05.416718 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 23984
I20260929 10:57:05.419358 24471 tablet_replica.cc:330] T cd4a4760cabd45deadab9a134668f8d1 P bde3d177c7c84afea69a699e117771f8: stopping tablet replica
I20260929 10:57:05.419480 24471 raft_consensus.cc:2229] T cd4a4760cabd45deadab9a134668f8d1 P bde3d177c7c84afea69a699e117771f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:57:05.419569 24471 raft_consensus.cc:2258] T cd4a4760cabd45deadab9a134668f8d1 P bde3d177c7c84afea69a699e117771f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:57:05.424228 24473 tablet_replica.cc:330] T cd4a4760cabd45deadab9a134668f8d1 P 1e4c5281b9df4fc58b8cc13dda2ccb61: stopping tablet replica
I20260929 10:57:05.424360 24473 raft_consensus.cc:2229] T cd4a4760cabd45deadab9a134668f8d1 P 1e4c5281b9df4fc58b8cc13dda2ccb61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:57:05.424475 24473 raft_consensus.cc:2258] T cd4a4760cabd45deadab9a134668f8d1 P 1e4c5281b9df4fc58b8cc13dda2ccb61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:57:05.424974 24473 ts_tablet_manager.cc:1905] T cd4a4760cabd45deadab9a134668f8d1 P 1e4c5281b9df4fc58b8cc13dda2ccb61: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 10:57:05.425575 23932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.238.193:34175 (error 108)
I20260929 10:57:05.426422 24441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:05.426312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:34175 (local address 127.0.0.1:33882)
0929 10:57:05.426358 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:05.426389 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.193:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260929 10:57:05.426530 23932 catalog_manager.cc:4584] TS 133a79f08d534b7f8fea4cac98dfce44 (127.10.238.193:34175): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cd4a4760cabd45deadab9a134668f8d1: Network error: Client connection negotiation failed: client connection to 127.10.238.193:34175: connect: Connection refused (error 111)
I20260929 10:57:05.426725 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 24148
I20260929 10:57:05.431811 24473 ts_tablet_manager.cc:1918] T cd4a4760cabd45deadab9a134668f8d1 P 1e4c5281b9df4fc58b8cc13dda2ccb61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:57:05.431900 24473 log.cc:1196] T cd4a4760cabd45deadab9a134668f8d1 P 1e4c5281b9df4fc58b8cc13dda2ccb61: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790679274510869-11195-0/minicluster-data/ts-2/wal/wals/cd4a4760cabd45deadab9a134668f8d1
I20260929 10:57:05.432181 24473 ts_tablet_manager.cc:1939] T cd4a4760cabd45deadab9a134668f8d1 P 1e4c5281b9df4fc58b8cc13dda2ccb61: Deleting consensus metadata
I20260929 10:57:05.432957 23930 catalog_manager.cc:4841] TS 1e4c5281b9df4fc58b8cc13dda2ccb61 (127.10.238.195:33633): tablet cd4a4760cabd45deadab9a134668f8d1 (table test-table [id=d99e97fa33834400a48efdd772db8776]) successfully deleted
I20260929 10:57:05.433578 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 24281
I20260929 10:57:05.441226 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 23913
I20260929 10:57:05.450734 11195 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 10:57:05.795367 11195 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 10:57:05.795492 11195 mini_ranger.cc:66] Stopping Ranger...
I20260929 10:57:06.140197 11195 mini_ranger.cc:68] Stopped Ranger
2026-09-29 10:57:06.140 UTC [22980] LOG: received smart shutdown request
2026-09-29 10:57:06.144 UTC [22980] LOG: background worker "logical replication launcher" (PID 22989) exited with exit code 1
2026-09-29 10:57:06.144 UTC [22984] LOG: shutting down
2026-09-29 10:57:06.318 UTC [22980] LOG: database system is shut down
2026-09-29T10:57:06Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47795 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:57:06 dist-test-slave-8nrt krb5kdc[24483](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:57:06 dist-test-slave-8nrt krb5kdc[24483](info): set up 2 sockets
Sep 29 10:57:06 dist-test-slave-8nrt krb5kdc[24483](info): commencing operation
krb5kdc: starting...
W20260929 10:57:07.465222 11195 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Sep 29 10:57:07 dist-test-slave-8nrt krb5kdc[24483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:57:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:57:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
I20260929 10:57:07.582203 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:42569
--webserver_interface=127.10.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:35295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:42569,127.10.238.253:45239,127.10.238.252:39815
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:57:07.661515 24496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:57:07.661717 24496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:57:07.661741 24496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:57:07.662926 24496 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:57:07.662967 24496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:57:07.662981 24496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:57:07.662994 24496 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:57:07.663051 24496 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:57:07.664546 24496 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:35295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:42569,127.10.238.253:45239,127.10.238.252:39815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:42569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:57:07.664815 24496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:57:07.665047 24496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:57:07.670099 24501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:07.670089 24502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:07.670089 24504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:57:07.670336 24496 server_base.cc:1034] running on GCE node
I20260929 10:57:07.670712 24496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:57:07.671062 24496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:57:07.672231 24496 hybrid_clock.cc:648] HybridClock initialized: now 1790679427672224 us; error 36 us; skew 500 ppm
Sep 29 10:57:07 dist-test-slave-8nrt krb5kdc[24483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679427, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:57:07.677518 24496 init.cc:377] Logged in from keytab as kudu/127.10.238.254@KRBTEST.COM (short username kudu)
I20260929 10:57:07.678017 24496 webserver.cc:460] Webserver started at http://127.10.238.254:41669/ using document root <none> and password file <none>
I20260929 10:57:07.678254 24496 fs_manager.cc:359] Metadata directory not provided
I20260929 10:57:07.678325 24496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:57:07.678464 24496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:57:07.679538 24496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/minicluster-data/master-0/data/instance:
uuid: "ddb9cd78ee4b4024b0066ad8ba658b14"
format_stamp: "Formatted at 2026-09-29 10:57:07 on dist-test-slave-8nrt"
server_key: "a30ef3fb7349fdec1e52d2d091e9512c"
server_key_iv: "e7be1028c95538258187566a7b70955e"
server_key_version: "encryptionkey@0"
I20260929 10:57:07.679912 24496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/minicluster-data/master-0/wal/instance:
uuid: "ddb9cd78ee4b4024b0066ad8ba658b14"
format_stamp: "Formatted at 2026-09-29 10:57:07 on dist-test-slave-8nrt"
server_key: "a30ef3fb7349fdec1e52d2d091e9512c"
server_key_iv: "e7be1028c95538258187566a7b70955e"
server_key_version: "encryptionkey@0"
I20260929 10:57:07.681380 24496 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 10:57:07.682215 24511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:07.682449 24496 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:57:07.682545 24496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "ddb9cd78ee4b4024b0066ad8ba658b14"
format_stamp: "Formatted at 2026-09-29 10:57:07 on dist-test-slave-8nrt"
server_key: "a30ef3fb7349fdec1e52d2d091e9512c"
server_key_iv: "e7be1028c95538258187566a7b70955e"
server_key_version: "encryptionkey@0"
I20260929 10:57:07.682602 24496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:57:07.698062 24496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:57:07.700351 24496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:57:07.700466 24496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:57:07.704385 24496 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.254:42569
I20260929 10:57:07.704456 24563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.254:42569 every 8 connection(s)
I20260929 10:57:07.704816 24496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:57:07.705808 24564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:57:07.706697 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 24496
I20260929 10:57:07.706825 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/minicluster-data/master-0/wal/instance
I20260929 10:57:07.707041 11195 external_mini_cluster.cc:1428] Setting key 8924d9d15963d7c63478f8fabbc37b06
I20260929 10:57:07.707540 24564 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.238.254" port: 42569 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 10:57:07 dist-test-slave-8nrt krb5kdc[24483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679427, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.238.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.253@KRBTEST.COM" created.
I20260929 10:57:07.717512 24567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:07.708259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48270 (local address 127.10.238.254:42569)
0929 10:57:07.708494 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:07.708502 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:07.708608 (+ 106us) server_negotiation.cc:407] Connection header received
0929 10:57:07.709135 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:07.709137 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:07.709157 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:07.709265 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:57:07.709846 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:07.710777 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:07.711675 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:07.712004 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:07.712133 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:07.714447 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:57:07.714455 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:57:07.714455 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:57:07.714462 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:57:07.715990 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:07.716355 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:07.716357 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:07.716357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:07.716446 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:07.716692 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:07.716693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:07.716694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:07.716911 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:57:07.717007 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:07.717188 (+ 181us) server_negotiation.cc:299] Negotiation successful
0929 10:57:07.717337 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":82,"threads_started":1}
I20260929 10:57:07.717648 24568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:07.708257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:42569 (local address 127.0.0.1:48270)
0929 10:57:07.708449 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:07.708493 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:07.708974 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:07.709298 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:57:07.709301 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:57:07.709583 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:57:07.709731 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:07.709735 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:07.710895 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:07.710897 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:07.711583 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:07.711585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:07.712106 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:07.712106 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:07.712171 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:07.712724 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:57:07.712732 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:57:07.714361 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:57:07.716086 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:07.716092 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:07.716094 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:07.716280 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:07.716520 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:07.716521 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:07.716521 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:07.716613 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:07.716985 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:57:07.716990 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:57:07.717051 (+ 61us) client_negotiation.cc:769] Sending connection context
0929 10:57:07.717162 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 10:57:07.717455 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":123,"thread_start_us":87,"threads_started":1}
I20260929 10:57:07.718585 24564 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45239 } has no permanent_uuid. Determining permanent_uuid...
I20260929 10:57:07.718927 24568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:07.718820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.253:45239 (local address 127.0.0.1:45798)
0929 10:57:07.718866 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:07.718908 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.253:45239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260929 10:57:07.719091 24512 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.238.253:45239: connect: Connection refused (error 111)
I20260929 10:57:07.719450 24568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:07.719388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.253:45239 (local address 127.0.0.1:45806)
0929 10:57:07.719413 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:07.719435 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.253:45239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260929 10:57:07.719565 24564 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.238.253:45239: Network error: Client connection negotiation failed: client connection to 127.10.238.253:45239: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.253.keytab.
Entry for principal kudu/127.10.238.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.253.keytab.
I20260929 10:57:07.731330 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.253:45239
--webserver_interface=127.10.238.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.253.keytab
--principal=kudu/127.10.238.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:35295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:42569,127.10.238.253:45239,127.10.238.252:39815
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:57:07.768807 24564 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45239 } attempt: 1
I20260929 10:57:07.769376 24568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:07.769199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.253:45239 (local address 127.0.0.1:45818)
0929 10:57:07.769292 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:07.769340 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.253:45239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 10:57:07.769757 24568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:07.769648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.253:45239 (local address 127.0.0.1:45820)
0929 10:57:07.769699 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:07.769726 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.253:45239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260929 10:57:07.769846 24564 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.238.253:45239: Network error: Client connection negotiation failed: client connection to 127.10.238.253:45239: connect: Connection refused (error 111)
W20260929 10:57:07.812223 24571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:57:07.812465 24571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:57:07.812502 24571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:57:07.813669 24571 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:57:07.813743 24571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:57:07.813767 24571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:57:07.813789 24571 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:57:07.813846 24571 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:57:07.815317 24571 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:35295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:42569,127.10.238.253:45239,127.10.238.252:39815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.253.keytab
--principal=kudu/127.10.238.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.253:45239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.238.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:57:07.815595 24571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:57:07.815833 24571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:57:07.820670 24577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:07.820680 24576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:07.820815 24579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:57:07.820956 24571 server_base.cc:1034] running on GCE node
I20260929 10:57:07.821214 24571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:57:07.821564 24571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:57:07.822739 24571 hybrid_clock.cc:648] HybridClock initialized: now 1790679427822710 us; error 49 us; skew 500 ppm
Sep 29 10:57:07 dist-test-slave-8nrt krb5kdc[24483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679427, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:57:07.825420 24571 init.cc:377] Logged in from keytab as kudu/127.10.238.253@KRBTEST.COM (short username kudu)
I20260929 10:57:07.825925 24571 webserver.cc:460] Webserver started at http://127.10.238.253:40907/ using document root <none> and password file <none>
I20260929 10:57:07.826154 24571 fs_manager.cc:359] Metadata directory not provided
I20260929 10:57:07.826222 24571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:57:07.826361 24571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:57:07.827411 24571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/minicluster-data/master-1/data/instance:
uuid: "5fd097b9153c4df890be21fc3a4282f1"
format_stamp: "Formatted at 2026-09-29 10:57:07 on dist-test-slave-8nrt"
server_key: "1e25b58ecfbf75d6cd8c014c4a3eafc0"
server_key_iv: "d785814ad59e9558df83bb4b16d1bc34"
server_key_version: "encryptionkey@0"
I20260929 10:57:07.827813 24571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/minicluster-data/master-1/wal/instance:
uuid: "5fd097b9153c4df890be21fc3a4282f1"
format_stamp: "Formatted at 2026-09-29 10:57:07 on dist-test-slave-8nrt"
server_key: "1e25b58ecfbf75d6cd8c014c4a3eafc0"
server_key_iv: "d785814ad59e9558df83bb4b16d1bc34"
server_key_version: "encryptionkey@0"
I20260929 10:57:07.829134 24571 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 10:57:07.829895 24586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:07.830077 24571 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:57:07.830175 24571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/minicluster-data/master-1/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/minicluster-data/master-1/wal
uuid: "5fd097b9153c4df890be21fc3a4282f1"
format_stamp: "Formatted at 2026-09-29 10:57:07 on dist-test-slave-8nrt"
server_key: "1e25b58ecfbf75d6cd8c014c4a3eafc0"
server_key_iv: "d785814ad59e9558df83bb4b16d1bc34"
server_key_version: "encryptionkey@0"
I20260929 10:57:07.830259 24571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:57:07.837980 24564 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45239 } attempt: 2
I20260929 10:57:07.838369 24568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:07.838246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.253:45239 (local address 127.0.0.1:45830)
0929 10:57:07.838304 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:07.838341 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.253:45239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260929 10:57:07.838702 24568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:07.838612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.253:45239 (local address 127.0.0.1:45846)
0929 10:57:07.838656 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:07.838679 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.253:45239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260929 10:57:07.838786 24564 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.238.253:45239: Network error: Client connection negotiation failed: client connection to 127.10.238.253:45239: connect: Connection refused (error 111)
I20260929 10:57:07.849139 24571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:57:07.850987 24571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:57:07.851148 24571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:57:07.855250 24571 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.253:45239
I20260929 10:57:07.855288 24638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.253:45239 every 8 connection(s)
I20260929 10:57:07.855640 24571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/minicluster-data/master-1/data/info.pb
I20260929 10:57:07.855882 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 24571
I20260929 10:57:07.856010 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/minicluster-data/master-1/wal/instance
I20260929 10:57:07.856201 11195 external_mini_cluster.cc:1428] Setting key 340f9fa4e5955ffce7a62b66601485ea
I20260929 10:57:07.856602 24639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:57:07.858178 24639 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.238.254" port: 42569 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.10.238.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.252@KRBTEST.COM" created.
Sep 29 10:57:07 dist-test-slave-8nrt krb5kdc[24483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679427, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.253@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:57:07.869664 24567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:07.858900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48272 (local address 127.10.238.254:42569)
0929 10:57:07.858963 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:07.858965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:07.859141 (+ 176us) server_negotiation.cc:407] Connection header received
0929 10:57:07.859668 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:07.859669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:07.859684 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:07.859729 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:57:07.860343 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:07.860825 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:07.861882 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:07.862091 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:07.862171 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:07.866611 (+ 4440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:57:07.866613 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:57:07.866613 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:57:07.866626 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:57:07.868218 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:07.868641 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:07.868643 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:07.868643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:07.868704 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:07.868961 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:07.868962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:07.868962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:07.869076 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:57:07.869151 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:07.869489 (+ 338us) server_negotiation.cc:299] Negotiation successful
0929 10:57:07.869576 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 10:57:07.869849 24642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:07.858867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:42569 (local address 127.0.0.1:48272)
0929 10:57:07.859022 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:07.859062 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:07.859562 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:07.859819 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:57:07.859822 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:57:07.860060 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:57:07.860241 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:07.860245 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:07.860924 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:07.860925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:07.861782 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:07.861785 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:07.862177 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:07.862179 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:07.862298 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:07.862813 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:57:07.862821 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:57:07.866502 (+ 3681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:57:07.868324 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:07.868332 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:07.868335 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:07.868560 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:07.868774 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:07.868774 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:07.868775 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:07.868890 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:07.869144 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:57:07.869151 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:57:07.869342 (+ 191us) client_negotiation.cc:769] Sending connection context
0929 10:57:07.869453 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 10:57:07.869606 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":77,"threads_started":1}
I20260929 10:57:07.870466 24639 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45239 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 10:57:07 dist-test-slave-8nrt krb5kdc[24483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679427, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.253@KRBTEST.COM for kudu/127.10.238.253@KRBTEST.COM
I20260929 10:57:07.878124 24642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:07.870724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.253:45239 (local address 127.0.0.1:45860)
0929 10:57:07.870774 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:07.870784 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:07.870824 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:07.871186 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:57:07.871186 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:57:07.871332 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:57:07.871377 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:07.871378 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:07.872224 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:07.872225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:07.872711 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:07.872712 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:07.873198 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:07.873199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:07.873298 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:07.873625 (+ 327us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:57:07.873629 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:57:07.875356 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:57:07.877009 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:07.877009 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:07.877010 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:07.877172 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:07.877436 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:07.877437 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:07.877438 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:07.877492 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:07.877778 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:57:07.877779 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:57:07.877814 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 10:57:07.877881 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 10:57:07.877981 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 10:57:07.878207 24644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:07.870893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45860 (local address 127.10.238.253:45239)
0929 10:57:07.871041 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:07.871045 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:07.871068 (+ 23us) server_negotiation.cc:407] Connection header received
0929 10:57:07.871094 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:07.871096 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:07.871106 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:07.871186 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:57:07.871452 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:07.872126 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:07.872791 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:07.873097 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:07.873202 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:07.875460 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:57:07.875468 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:57:07.875468 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:57:07.875475 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:57:07.876906 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:07.877291 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:07.877295 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:07.877295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:07.877351 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:07.877565 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:07.877566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:07.877566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:07.877707 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:57:07.877808 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:07.877925 (+ 117us) server_negotiation.cc:299] Negotiation successful
0929 10:57:07.878053 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":122,"thread_start_us":93,"threads_started":1}
I20260929 10:57:07.878851 24639 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.238.252" port: 39815 } has no permanent_uuid. Determining permanent_uuid...
I20260929 10:57:07.879226 24642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:07.879128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.252:39815 (local address 127.0.0.1:60782)
0929 10:57:07.879159 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:07.879207 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.252:39815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260929 10:57:07.879397 24587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.238.252:39815: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.252.keytab.
Entry for principal kudu/127.10.238.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.252.keytab.
I20260929 10:57:07.879921 24642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:07.879806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.252:39815 (local address 127.0.0.1:60792)
0929 10:57:07.879856 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:07.879888 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.252:39815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260929 10:57:07.880061 24639 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.238.252:39815: Network error: Client connection negotiation failed: client connection to 127.10.238.252:39815: connect: Connection refused (error 111)
I20260929 10:57:07.880388 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.252:39815
--webserver_interface=127.10.238.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.252.keytab
--principal=kudu/127.10.238.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:35295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:42569,127.10.238.253:45239,127.10.238.252:39815
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:57:07.929328 24639 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.238.252" port: 39815 } attempt: 1
I20260929 10:57:07.929813 24642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:07.929676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.252:39815 (local address 127.0.0.1:60806)
0929 10:57:07.929740 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:07.929781 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.252:39815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 10:57:07.929920 24564 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45239 } attempt: 3
I20260929 10:57:07.930258 24642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:07.930131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.252:39815 (local address 127.0.0.1:60816)
0929 10:57:07.930196 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:07.930227 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.252:39815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 10:57:07.930404 24639 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.238.252:39815: Network error: Client connection negotiation failed: client connection to 127.10.238.252:39815: connect: Connection refused (error 111)
Sep 29 10:57:07 dist-test-slave-8nrt krb5kdc[24483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679427, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for kudu/127.10.238.253@KRBTEST.COM
I20260929 10:57:07.936954 24568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:07.930205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.253:45239 (local address 127.0.0.1:45868)
0929 10:57:07.930252 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:07.930262 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:07.930319 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:07.930513 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:57:07.930514 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:57:07.930678 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:57:07.930743 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:07.930745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:07.931475 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:07.931476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:07.932002 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:07.932003 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:07.932392 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:07.932393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:07.932505 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:07.932972 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:57:07.932990 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:57:07.934308 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:57:07.936030 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:07.936031 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:07.936031 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:07.936193 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:07.936406 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:07.936407 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:07.936407 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:07.936459 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:07.936723 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:57:07.936723 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:57:07.936762 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 10:57:07.936813 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 10:57:07.936888 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 10:57:07.937011 24644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:07.930257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45868 (local address 127.10.238.253:45239)
0929 10:57:07.930295 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:07.930297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:07.930332 (+ 35us) server_negotiation.cc:407] Connection header received
0929 10:57:07.930411 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:07.930412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:07.930425 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:07.930505 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:57:07.930848 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:07.931379 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:07.932088 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:07.932314 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:07.932393 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:07.934389 (+ 1996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:57:07.934391 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:57:07.934391 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:57:07.934399 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:57:07.935937 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:07.936266 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:07.936267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:07.936268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:07.936332 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:07.936520 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:07.936521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:07.936521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:07.936660 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:57:07.936729 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:07.936839 (+ 110us) server_negotiation.cc:299] Negotiation successful
0929 10:57:07.936922 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 10:57:07.937489 24564 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.238.252" port: 39815 } has no permanent_uuid. Determining permanent_uuid...
I20260929 10:57:07.937829 24568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:07.937722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.252:39815 (local address 127.0.0.1:60828)
0929 10:57:07.937772 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:07.937798 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.252:39815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 10:57:07.938153 24568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:07.938064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.252:39815 (local address 127.0.0.1:60830)
0929 10:57:07.938109 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:07.938132 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.252:39815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260929 10:57:07.938272 24564 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.238.252:39815: Network error: Client connection negotiation failed: client connection to 127.10.238.252:39815: connect: Connection refused (error 111)
W20260929 10:57:07.963317 24646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:57:07.963505 24646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:57:07.963526 24646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:57:07.964573 24646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:57:07.964607 24646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:57:07.964622 24646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:57:07.964633 24646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:57:07.964645 24646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:57:07.966051 24646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:35295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:42569,127.10.238.253:45239,127.10.238.252:39815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.252.keytab
--principal=kudu/127.10.238.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.252:39815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.238.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:57:07.966233 24646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:57:07.966394 24646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 10:57:07.969415 24564 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.238.252" port: 39815 } attempt: 1
I20260929 10:57:07.969861 24568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:07.969720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.252:39815 (local address 127.0.0.1:60836)
0929 10:57:07.969791 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:07.969832 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.252:39815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 10:57:07.970245 24568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:07.970142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.252:39815 (local address 127.0.0.1:60848)
0929 10:57:07.970193 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:07.970217 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.252:39815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260929 10:57:07.970424 24564 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.238.252:39815: Network error: Client connection negotiation failed: client connection to 127.10.238.252:39815: connect: Connection refused (error 111)
W20260929 10:57:07.971280 24651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:07.971290 24652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:07.971290 24654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:57:07.971488 24646 server_base.cc:1034] running on GCE node
I20260929 10:57:07.971727 24646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:57:07.971993 24646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:57:07.973172 24646 hybrid_clock.cc:648] HybridClock initialized: now 1790679427973181 us; error 50 us; skew 500 ppm
Sep 29 10:57:07 dist-test-slave-8nrt krb5kdc[24483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679427, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:57:07.975705 24646 init.cc:377] Logged in from keytab as kudu/127.10.238.252@KRBTEST.COM (short username kudu)
I20260929 10:57:07.976122 24646 webserver.cc:460] Webserver started at http://127.10.238.252:46597/ using document root <none> and password file <none>
I20260929 10:57:07.976358 24646 fs_manager.cc:359] Metadata directory not provided
I20260929 10:57:07.976440 24646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:57:07.976585 24646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:57:07.977679 24646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/minicluster-data/master-2/data/instance:
uuid: "46f1c841d0944835a4c09c4f9c0542a4"
format_stamp: "Formatted at 2026-09-29 10:57:07 on dist-test-slave-8nrt"
server_key: "1f4d09816767322fbd3be4696642a595"
server_key_iv: "af5d197456352aa456d2e4655df10454"
server_key_version: "encryptionkey@0"
I20260929 10:57:07.978045 24646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/minicluster-data/master-2/wal/instance:
uuid: "46f1c841d0944835a4c09c4f9c0542a4"
format_stamp: "Formatted at 2026-09-29 10:57:07 on dist-test-slave-8nrt"
server_key: "1f4d09816767322fbd3be4696642a595"
server_key_iv: "af5d197456352aa456d2e4655df10454"
server_key_version: "encryptionkey@0"
I20260929 10:57:07.979336 24646 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 10:57:07.980017 24661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:07.980183 24646 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:57:07.980278 24646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/minicluster-data/master-2/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/minicluster-data/master-2/wal
uuid: "46f1c841d0944835a4c09c4f9c0542a4"
format_stamp: "Formatted at 2026-09-29 10:57:07 on dist-test-slave-8nrt"
server_key: "1f4d09816767322fbd3be4696642a595"
server_key_iv: "af5d197456352aa456d2e4655df10454"
server_key_version: "encryptionkey@0"
I20260929 10:57:07.980361 24646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:57:07.990155 24646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:57:07.992354 24646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:57:07.992513 24646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:57:07.996714 24646 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.252:39815
I20260929 10:57:07.996784 24713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.252:39815 every 8 connection(s)
I20260929 10:57:07.997119 24646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/minicluster-data/master-2/data/info.pb
I20260929 10:57:07.998014 24714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:57:07.998567 24639 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.238.252" port: 39815 } attempt: 2
I20260929 10:57:07.999745 24714 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.238.254" port: 42569 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 10:57:07 dist-test-slave-8nrt krb5kdc[24483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679427, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.253@KRBTEST.COM for kudu/127.10.238.252@KRBTEST.COM
Sep 29 10:57:07 dist-test-slave-8nrt krb5kdc[24483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679427, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.252@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:57:08.005098 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 24646
I20260929 10:57:08.005290 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790679274510869-11195-0/minicluster-data/master-2/wal/instance
I20260929 10:57:08.005540 11195 external_mini_cluster.cc:1428] Setting key 356723ab4d4d18059711ce434c688fbf
I20260929 10:57:08.008328 24642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:07.998892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.252:39815 (local address 127.0.0.1:60858)
0929 10:57:07.998965 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:07.998979 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:07.999039 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:07.999894 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:57:07.999896 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:57:08.000066 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:57:08.000119 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:08.000121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:08.001365 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:08.001366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:08.001866 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:08.001868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:08.002487 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:08.002488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:08.002547 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:08.003007 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:57:08.003013 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:57:08.004572 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:57:08.006915 (+ 2343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:08.006916 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:08.006917 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:08.007153 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:08.007423 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:08.007424 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:08.007425 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:08.007493 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:08.008005 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:57:08.008006 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:57:08.008054 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 10:57:08.008127 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 10:57:08.008233 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 10:57:08.008581 24716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:07.999017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60858 (local address 127.10.238.252:39815)
0929 10:57:07.999203 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:07.999219 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:07.999250 (+ 31us) server_negotiation.cc:407] Connection header received
0929 10:57:07.999707 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:07.999713 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:07.999760 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:07.999872 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:57:08.000242 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:08.001247 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:08.001973 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:08.002394 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:08.002520 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:08.004664 (+ 2144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:57:08.004672 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:57:08.004673 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:57:08.004689 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:57:08.006791 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:08.007237 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:08.007239 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:08.007240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:08.007342 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:08.007563 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:08.007564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:08.007564 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:08.007891 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:57:08.007976 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:08.008214 (+ 238us) server_negotiation.cc:299] Negotiation successful
0929 10:57:08.008345 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":91,"threads_started":1}
I20260929 10:57:08.008821 24567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:08.000476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48282 (local address 127.10.238.254:42569)
0929 10:57:08.000521 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:08.000522 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:08.000678 (+ 156us) server_negotiation.cc:407] Connection header received
0929 10:57:08.000756 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:08.000757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:08.000768 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:08.000838 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:57:08.001348 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:08.001821 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:08.002670 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:08.002897 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:08.002985 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:08.005306 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:57:08.005308 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:57:08.005308 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:57:08.005316 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:57:08.007538 (+ 2222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:08.007923 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:08.007924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:08.007924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:08.007991 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:08.008220 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:08.008222 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:08.008222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:08.008351 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:57:08.008397 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:08.008650 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 10:57:08.008735 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 10:57:08.009014 24717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:08.000419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:42569 (local address 127.0.0.1:48282)
0929 10:57:08.000585 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:08.000616 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:08.000692 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:08.000857 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:57:08.000859 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:57:08.001113 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:57:08.001202 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:08.001207 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:08.001910 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:08.001912 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:08.002579 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:08.002582 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:08.002999 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:08.003000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:08.003073 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:08.003501 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:57:08.003509 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:57:08.005171 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:57:08.007631 (+ 2460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:08.007639 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:08.007641 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:08.007844 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:08.008051 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:08.008052 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:08.008052 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:08.008121 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:08.008451 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:57:08.008458 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:57:08.008506 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 10:57:08.008608 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 10:57:08.008720 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":76,"threads_started":1}
I20260929 10:57:08.009475 24714 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45239 } has no permanent_uuid. Determining permanent_uuid...
I20260929 10:57:08.010524 24639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fd097b9153c4df890be21fc3a4282f1: Bootstrap starting.
I20260929 10:57:08.011540 24639 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fd097b9153c4df890be21fc3a4282f1: Neither blocks nor log segments found. Creating new log.
I20260929 10:57:08.011839 24639 log.cc:824] T 00000000000000000000000000000000 P 5fd097b9153c4df890be21fc3a4282f1: Log is configured to *not* fsync() on all Append() calls
I20260929 10:57:08.012682 24639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fd097b9153c4df890be21fc3a4282f1: No bootstrap required, opened a new log
Sep 29 10:57:08 dist-test-slave-8nrt krb5kdc[24483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.238.252@KRBTEST.COM
Sep 29 10:57:08 dist-test-slave-8nrt krb5kdc[24483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679427, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.252@KRBTEST.COM for kudu/127.10.238.253@KRBTEST.COM
I20260929 10:57:08.014744 24639 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fd097b9153c4df890be21fc3a4282f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb9cd78ee4b4024b0066ad8ba658b14" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 42569 } } peers { permanent_uuid: "5fd097b9153c4df890be21fc3a4282f1" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45239 } } peers { permanent_uuid: "46f1c841d0944835a4c09c4f9c0542a4" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 39815 } }
I20260929 10:57:08.014889 24639 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5fd097b9153c4df890be21fc3a4282f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:57:08.014925 24639 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fd097b9153c4df890be21fc3a4282f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fd097b9153c4df890be21fc3a4282f1, State: Initialized, Role: FOLLOWER
I20260929 10:57:08.015059 24639 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fd097b9153c4df890be21fc3a4282f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb9cd78ee4b4024b0066ad8ba658b14" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 42569 } } peers { permanent_uuid: "5fd097b9153c4df890be21fc3a4282f1" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45239 } } peers { permanent_uuid: "46f1c841d0944835a4c09c4f9c0542a4" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 39815 } }
I20260929 10:57:08.015379 24639 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fd097b9153c4df890be21fc3a4282f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:57:08.015789 24722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fd097b9153c4df890be21fc3a4282f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb9cd78ee4b4024b0066ad8ba658b14" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 42569 } } peers { permanent_uuid: "5fd097b9153c4df890be21fc3a4282f1" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45239 } } peers { permanent_uuid: "46f1c841d0944835a4c09c4f9c0542a4" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 39815 } } }
I20260929 10:57:08.015890 24722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fd097b9153c4df890be21fc3a4282f1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:57:08.016471 24719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:08.005901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60872 (local address 127.10.238.252:39815)
0929 10:57:08.006967 (+ 1066us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:08.006969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:08.007136 (+ 167us) server_negotiation.cc:407] Connection header received
0929 10:57:08.007191 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:08.007192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:08.007203 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:08.007274 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:57:08.007986 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:08.009461 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:08.010405 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:08.010627 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:08.010666 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:08.013248 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:57:08.013249 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:57:08.013250 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:57:08.013279 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:57:08.015009 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:08.015431 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:08.015432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:08.015432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:08.015502 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:08.015729 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:08.015731 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:08.015731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:08.015876 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:57:08.016015 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:08.016235 (+ 220us) server_negotiation.cc:299] Negotiation successful
0929 10:57:08.016320 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1045,"thread_start_us":110,"threads_started":1}
W20260929 10:57:08.017604 24733 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5fd097b9153c4df890be21fc3a4282f1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:57:08.017663 24733 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:57:08.017721 24720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:08.009746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.253:45239 (local address 127.0.0.1:45876)
0929 10:57:08.009893 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:08.009904 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:08.009950 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:08.010124 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:57:08.010124 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:57:08.010298 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:57:08.010349 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:08.010351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:08.011240 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:08.011241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:08.012039 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:08.012042 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:08.012504 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:08.012506 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:08.012578 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:08.013080 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:57:08.013085 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:57:08.014467 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:57:08.016556 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:08.016557 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:08.016558 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:08.016775 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:08.016962 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:08.016963 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:08.016963 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:08.017030 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:08.017409 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:57:08.017410 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:57:08.017455 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 10:57:08.017520 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 10:57:08.017620 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":75,"threads_started":1}
I20260929 10:57:08.018091 24644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:08.009905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45876 (local address 127.10.238.253:45239)
0929 10:57:08.009943 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:08.009944 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:08.009952 (+ 8us) server_negotiation.cc:407] Connection header received
0929 10:57:08.010039 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:08.010040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:08.010050 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:08.010111 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:57:08.010432 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:08.011133 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:08.012145 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:08.012409 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:08.012476 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:08.014555 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:57:08.014557 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:57:08.014557 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:57:08.014564 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:57:08.016345 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:08.016836 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:08.016837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:08.016838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:08.016897 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:08.017114 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:08.017115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:08.017115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:08.017311 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:57:08.017379 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:08.017934 (+ 555us) server_negotiation.cc:299] Negotiation successful
0929 10:57:08.018011 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 10:57:08.018400 24714 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.238.252" port: 39815 } has no permanent_uuid. Determining permanent_uuid...
I20260929 10:57:08.018103 24571 master_runner.cc:186] Error getting master registration for 127.10.238.254:42569: OK, instance_id { permanent_uuid: "ddb9cd78ee4b4024b0066ad8ba658b14" instance_seqno: 1790679427703319 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 10:57:08.020576 24571 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 29 10:57:08 dist-test-slave-8nrt krb5kdc[24483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679427, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.252@KRBTEST.COM for kudu/127.10.238.252@KRBTEST.COM
Sep 29 10:57:08 dist-test-slave-8nrt krb5kdc[24483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:57:08.025966 24720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:08.018608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.252:39815 (local address 127.0.0.1:60882)
0929 10:57:08.018640 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:08.018649 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:08.018687 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:08.018829 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:57:08.018829 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:57:08.018972 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:57:08.019027 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:08.019028 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:08.019874 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:08.019875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:08.020598 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:08.020601 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:08.021047 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:08.021048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:08.021104 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:08.021525 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:57:08.021530 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:57:08.023062 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:57:08.024834 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:08.024835 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:08.024835 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:08.025019 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:08.025199 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:08.025200 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:08.025201 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:08.025275 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:08.025646 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:57:08.025646 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:57:08.025683 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 10:57:08.025765 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 10:57:08.025864 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260929 10:57:08.026021 24719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:08.018663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60882 (local address 127.10.238.252:39815)
0929 10:57:08.018698 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:08.018699 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:08.018706 (+ 7us) server_negotiation.cc:407] Connection header received
0929 10:57:08.018745 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:08.018746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:08.018756 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:08.018798 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:57:08.019112 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:08.019790 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:08.020677 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:08.020977 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:08.021048 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:08.023163 (+ 2115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:57:08.023164 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:57:08.023164 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:57:08.023171 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:57:08.024748 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:08.025090 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:08.025092 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:08.025092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:08.025148 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:08.025449 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:08.025450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:08.025450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:08.025564 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:57:08.025641 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:08.025786 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 10:57:08.025915 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":18}
I20260929 10:57:08.026472 24567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:08.017407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48294 (local address 127.10.238.254:42569)
0929 10:57:08.017512 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:08.017514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:08.017522 (+ 8us) server_negotiation.cc:407] Connection header received
0929 10:57:08.017545 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:08.017545 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:08.017556 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:08.017606 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:57:08.018234 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:08.018939 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:08.020038 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:08.020309 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:08.020372 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:08.023220 (+ 2848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:57:08.023221 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:57:08.023221 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:57:08.023227 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:57:08.024786 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:08.025153 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:08.025154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:08.025154 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:08.025218 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:08.025448 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:08.025449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:08.025450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:08.025579 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:57:08.025749 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:08.026307 (+ 558us) server_negotiation.cc:299] Negotiation successful
0929 10:57:08.026392 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 10:57:08.027686 24714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46f1c841d0944835a4c09c4f9c0542a4: Bootstrap starting.
I20260929 10:57:08.028314 24714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46f1c841d0944835a4c09c4f9c0542a4: Neither blocks nor log segments found. Creating new log.
I20260929 10:57:08.028638 24714 log.cc:824] T 00000000000000000000000000000000 P 46f1c841d0944835a4c09c4f9c0542a4: Log is configured to *not* fsync() on all Append() calls
I20260929 10:57:08.029481 24714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46f1c841d0944835a4c09c4f9c0542a4: No bootstrap required, opened a new log
Sep 29 10:57:08 dist-test-slave-8nrt krb5kdc[24483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.238.253@KRBTEST.COM
I20260929 10:57:08.031733 24714 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46f1c841d0944835a4c09c4f9c0542a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb9cd78ee4b4024b0066ad8ba658b14" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 42569 } } peers { permanent_uuid: "5fd097b9153c4df890be21fc3a4282f1" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45239 } } peers { permanent_uuid: "46f1c841d0944835a4c09c4f9c0542a4" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 39815 } }
I20260929 10:57:08.031867 24714 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46f1c841d0944835a4c09c4f9c0542a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:57:08.031888 24714 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46f1c841d0944835a4c09c4f9c0542a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46f1c841d0944835a4c09c4f9c0542a4, State: Initialized, Role: FOLLOWER
I20260929 10:57:08.032009 24714 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46f1c841d0944835a4c09c4f9c0542a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb9cd78ee4b4024b0066ad8ba658b14" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 42569 } } peers { permanent_uuid: "5fd097b9153c4df890be21fc3a4282f1" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45239 } } peers { permanent_uuid: "46f1c841d0944835a4c09c4f9c0542a4" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 39815 } }
I20260929 10:57:08.032341 24736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46f1c841d0944835a4c09c4f9c0542a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb9cd78ee4b4024b0066ad8ba658b14" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 42569 } } peers { permanent_uuid: "5fd097b9153c4df890be21fc3a4282f1" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45239 } } peers { permanent_uuid: "46f1c841d0944835a4c09c4f9c0542a4" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 39815 } } }
I20260929 10:57:08.032444 24736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46f1c841d0944835a4c09c4f9c0542a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:57:08.032478 24714 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46f1c841d0944835a4c09c4f9c0542a4 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 10:57:08.034516 24747 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 46f1c841d0944835a4c09c4f9c0542a4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:57:08.034623 24747 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:57:08.035687 24644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:08.027154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45884 (local address 127.10.238.253:45239)
0929 10:57:08.027186 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:08.027187 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:08.027195 (+ 8us) server_negotiation.cc:407] Connection header received
0929 10:57:08.027246 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:08.027246 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:08.027254 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:08.027291 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:57:08.027879 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:08.028302 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:08.029200 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:08.029482 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:08.029557 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:08.032251 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:57:08.032253 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:57:08.032253 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:57:08.032260 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:57:08.034197 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:08.034601 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:08.034603 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:08.034603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:08.034662 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:08.034865 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:08.034866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:08.034866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:08.034991 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:57:08.035154 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:08.035525 (+ 371us) server_negotiation.cc:299] Negotiation successful
0929 10:57:08.035596 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260929 10:57:08.034868 24646 master_runner.cc:186] Error getting master registration for 127.10.238.254:42569: OK, instance_id { permanent_uuid: "ddb9cd78ee4b4024b0066ad8ba658b14" instance_seqno: 1790679427703319 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 10:57:08.037216 11195 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:57:08.037317 24646 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 10:57:08.045667 24564 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.238.252" port: 39815 } attempt: 2
I20260929 10:57:08.050690 24719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:08.038746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60890 (local address 127.10.238.252:39815)
0929 10:57:08.038783 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:08.038785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:08.041746 (+ 2961us) server_negotiation.cc:407] Connection header received
0929 10:57:08.041835 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:08.041836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:08.041846 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:08.041876 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:57:08.042584 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:08.043258 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:08.044356 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:08.044658 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:08.044694 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:08.047582 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:57:08.047583 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:57:08.047584 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:57:08.047590 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:57:08.049321 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:08.049714 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:08.049715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:08.049716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:08.049779 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:08.050014 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:08.050015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:08.050016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:08.050149 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:57:08.050225 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:08.050489 (+ 264us) server_negotiation.cc:299] Negotiation successful
0929 10:57:08.050576 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 10:57:08.051041 24644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:08.038830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45898 (local address 127.10.238.253:45239)
0929 10:57:08.038864 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:08.038865 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:08.041464 (+ 2599us) server_negotiation.cc:407] Connection header received
0929 10:57:08.041530 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:08.041531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:08.041542 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:08.041597 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:57:08.042463 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:08.043197 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:08.044354 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:08.044641 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:08.044683 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:08.046899 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:57:08.046900 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:57:08.046901 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:57:08.046908 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:57:08.049313 (+ 2405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:08.050134 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:08.050135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:08.050136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:08.050188 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:08.050395 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:08.050396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:08.050396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:08.050532 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:57:08.050609 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:08.050839 (+ 230us) server_negotiation.cc:299] Negotiation successful
0929 10:57:08.050919 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
Sep 29 10:57:08 dist-test-slave-8nrt krb5kdc[24483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679427, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for kudu/127.10.238.252@KRBTEST.COM
I20260929 10:57:08.055114 24568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:08.045936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.252:39815 (local address 127.0.0.1:60898)
0929 10:57:08.046044 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:08.046054 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:08.046095 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:08.046265 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:57:08.046267 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:57:08.046437 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:57:08.046475 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:08.046477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:08.047439 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:08.047440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:08.048221 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:08.048223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:08.049377 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:08.049378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:08.049438 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:08.050303 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:57:08.050307 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:57:08.052468 (+ 2161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:57:08.054078 (+ 1610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:08.054079 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:08.054080 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:08.054248 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:08.054471 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:08.054472 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:08.054472 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:08.054535 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:08.054789 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:57:08.054790 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:57:08.054835 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 10:57:08.054918 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 10:57:08.055016 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260929 10:57:08.055124 24716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:08.046062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60898 (local address 127.10.238.252:39815)
0929 10:57:08.046104 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:08.046106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:08.046117 (+ 11us) server_negotiation.cc:407] Connection header received
0929 10:57:08.046140 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:08.046141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:08.046150 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:08.046221 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:57:08.046551 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:08.047338 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:08.048550 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:08.048882 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:08.048946 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:08.052558 (+ 3612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:57:08.052559 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:57:08.052559 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:57:08.052565 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:57:08.053973 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:08.054341 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:08.054342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:08.054343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:08.054398 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:08.054602 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:08.054603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:08.054603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:08.054721 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:57:08.054803 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:08.054942 (+ 139us) server_negotiation.cc:299] Negotiation successful
0929 10:57:08.055018 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 10:57:08.056262 24567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:08.038553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48298 (local address 127.10.238.254:42569)
0929 10:57:08.038604 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:08.038606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:08.041613 (+ 3007us) server_negotiation.cc:407] Connection header received
0929 10:57:08.041687 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:08.041688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:08.041700 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:08.041753 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:57:08.042439 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:08.043193 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:08.044275 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:08.044575 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:08.044615 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:08.048315 (+ 3700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:57:08.048316 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:57:08.048317 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:57:08.048324 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:57:08.050198 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:08.052869 (+ 2671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:08.052870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:08.052871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:08.052935 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:08.055667 (+ 2732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:08.055668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:08.055668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:08.055811 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:57:08.055900 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:08.056114 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 10:57:08.056185 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 10:57:08.057013 24564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddb9cd78ee4b4024b0066ad8ba658b14: Bootstrap starting.
I20260929 10:57:08.057734 24564 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ddb9cd78ee4b4024b0066ad8ba658b14: Neither blocks nor log segments found. Creating new log.
I20260929 10:57:08.058039 24564 log.cc:824] T 00000000000000000000000000000000 P ddb9cd78ee4b4024b0066ad8ba658b14: Log is configured to *not* fsync() on all Append() calls
I20260929 10:57:08.058854 24564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddb9cd78ee4b4024b0066ad8ba658b14: No bootstrap required, opened a new log
I20260929 10:57:08.061092 24564 raft_consensus.cc:348] T 00000000000000000000000000000000 P ddb9cd78ee4b4024b0066ad8ba658b14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb9cd78ee4b4024b0066ad8ba658b14" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 42569 } } peers { permanent_uuid: "5fd097b9153c4df890be21fc3a4282f1" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45239 } } peers { permanent_uuid: "46f1c841d0944835a4c09c4f9c0542a4" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 39815 } }
I20260929 10:57:08.061228 24564 raft_consensus.cc:374] T 00000000000000000000000000000000 P ddb9cd78ee4b4024b0066ad8ba658b14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:57:08.061275 24564 raft_consensus.cc:729] T 00000000000000000000000000000000 P ddb9cd78ee4b4024b0066ad8ba658b14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddb9cd78ee4b4024b0066ad8ba658b14, State: Initialized, Role: FOLLOWER
I20260929 10:57:08.061432 24564 consensus_queue.cc:260] T 00000000000000000000000000000000 P ddb9cd78ee4b4024b0066ad8ba658b14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb9cd78ee4b4024b0066ad8ba658b14" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 42569 } } peers { permanent_uuid: "5fd097b9153c4df890be21fc3a4282f1" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45239 } } peers { permanent_uuid: "46f1c841d0944835a4c09c4f9c0542a4" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 39815 } }
I20260929 10:57:08.061713 24757 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddb9cd78ee4b4024b0066ad8ba658b14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb9cd78ee4b4024b0066ad8ba658b14" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 42569 } } peers { permanent_uuid: "5fd097b9153c4df890be21fc3a4282f1" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45239 } } peers { permanent_uuid: "46f1c841d0944835a4c09c4f9c0542a4" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 39815 } } }
I20260929 10:57:08.061780 24564 sys_catalog.cc:564] T 00000000000000000000000000000000 P ddb9cd78ee4b4024b0066ad8ba658b14 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:57:08.061856 24757 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddb9cd78ee4b4024b0066ad8ba658b14 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 10:57:08.063568 24768 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ddb9cd78ee4b4024b0066ad8ba658b14: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:57:08.063674 24768 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:57:08.066290 24496 master_runner.cc:263] No leader master found from master ddb9cd78ee4b4024b0066ad8ba658b14
I20260929 10:57:08.066337 24496 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 10:57:08.072158 24736 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46f1c841d0944835a4c09c4f9c0542a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:57:08.072293 24736 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46f1c841d0944835a4c09c4f9c0542a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb9cd78ee4b4024b0066ad8ba658b14" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 42569 } } peers { permanent_uuid: "5fd097b9153c4df890be21fc3a4282f1" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45239 } } peers { permanent_uuid: "46f1c841d0944835a4c09c4f9c0542a4" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 39815 } }
I20260929 10:57:08.072552 24736 leader_election.cc:288] T 00000000000000000000000000000000 P 46f1c841d0944835a4c09c4f9c0542a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddb9cd78ee4b4024b0066ad8ba658b14 (127.10.238.254:42569), 5fd097b9153c4df890be21fc3a4282f1 (127.10.238.253:45239)
I20260929 10:57:08.072866 24538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "46f1c841d0944835a4c09c4f9c0542a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddb9cd78ee4b4024b0066ad8ba658b14" is_pre_election: true
I20260929 10:57:08.072872 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "46f1c841d0944835a4c09c4f9c0542a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fd097b9153c4df890be21fc3a4282f1" is_pre_election: true
I20260929 10:57:08.073005 24538 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ddb9cd78ee4b4024b0066ad8ba658b14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46f1c841d0944835a4c09c4f9c0542a4 in term 0.
I20260929 10:57:08.073028 24613 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5fd097b9153c4df890be21fc3a4282f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46f1c841d0944835a4c09c4f9c0542a4 in term 0.
I20260929 10:57:08.073585 24662 leader_election.cc:302] T 00000000000000000000000000000000 P 46f1c841d0944835a4c09c4f9c0542a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46f1c841d0944835a4c09c4f9c0542a4, 5fd097b9153c4df890be21fc3a4282f1; no voters:
I20260929 10:57:08.073788 24736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46f1c841d0944835a4c09c4f9c0542a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:57:08.073853 24736 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46f1c841d0944835a4c09c4f9c0542a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:57:08.073879 24736 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46f1c841d0944835a4c09c4f9c0542a4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:57:08.074949 24736 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46f1c841d0944835a4c09c4f9c0542a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb9cd78ee4b4024b0066ad8ba658b14" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 42569 } } peers { permanent_uuid: "5fd097b9153c4df890be21fc3a4282f1" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45239 } } peers { permanent_uuid: "46f1c841d0944835a4c09c4f9c0542a4" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 39815 } }
I20260929 10:57:08.075088 24736 leader_election.cc:288] T 00000000000000000000000000000000 P 46f1c841d0944835a4c09c4f9c0542a4 [CANDIDATE]: Term 1 election: Requested vote from peers ddb9cd78ee4b4024b0066ad8ba658b14 (127.10.238.254:42569), 5fd097b9153c4df890be21fc3a4282f1 (127.10.238.253:45239)
I20260929 10:57:08.075268 24538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "46f1c841d0944835a4c09c4f9c0542a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddb9cd78ee4b4024b0066ad8ba658b14"
I20260929 10:57:08.075344 24538 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ddb9cd78ee4b4024b0066ad8ba658b14 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:57:08.075318 24613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "46f1c841d0944835a4c09c4f9c0542a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fd097b9153c4df890be21fc3a4282f1"
I20260929 10:57:08.075407 24613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fd097b9153c4df890be21fc3a4282f1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:57:08.076282 24613 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5fd097b9153c4df890be21fc3a4282f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46f1c841d0944835a4c09c4f9c0542a4 in term 1.
I20260929 10:57:08.076464 24662 leader_election.cc:302] T 00000000000000000000000000000000 P 46f1c841d0944835a4c09c4f9c0542a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46f1c841d0944835a4c09c4f9c0542a4, 5fd097b9153c4df890be21fc3a4282f1; no voters:
I20260929 10:57:08.076552 24736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46f1c841d0944835a4c09c4f9c0542a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:57:08.076548 24538 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ddb9cd78ee4b4024b0066ad8ba658b14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46f1c841d0944835a4c09c4f9c0542a4 in term 1.
I20260929 10:57:08.076707 24736 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46f1c841d0944835a4c09c4f9c0542a4 [term 1 LEADER]: Becoming Leader. State: Replica: 46f1c841d0944835a4c09c4f9c0542a4, State: Running, Role: LEADER
I20260929 10:57:08.076798 24736 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46f1c841d0944835a4c09c4f9c0542a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb9cd78ee4b4024b0066ad8ba658b14" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 42569 } } peers { permanent_uuid: "5fd097b9153c4df890be21fc3a4282f1" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45239 } } peers { permanent_uuid: "46f1c841d0944835a4c09c4f9c0542a4" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 39815 } }
I20260929 10:57:08.077201 24736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46f1c841d0944835a4c09c4f9c0542a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46f1c841d0944835a4c09c4f9c0542a4. Latest consensus state: current_term: 1 leader_uuid: "46f1c841d0944835a4c09c4f9c0542a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb9cd78ee4b4024b0066ad8ba658b14" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 42569 } } peers { permanent_uuid: "5fd097b9153c4df890be21fc3a4282f1" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45239 } } peers { permanent_uuid: "46f1c841d0944835a4c09c4f9c0542a4" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 39815 } } }
I20260929 10:57:08.077292 24736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46f1c841d0944835a4c09c4f9c0542a4 [sys.catalog]: This master's current role is: LEADER
I20260929 10:57:08.081446 24772 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:57:08.081732 24772 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:57:08.083645 24538 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ddb9cd78ee4b4024b0066ad8ba658b14 [term 1 FOLLOWER]: Refusing update from remote peer 46f1c841d0944835a4c09c4f9c0542a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:57:08.083685 24613 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5fd097b9153c4df890be21fc3a4282f1 [term 1 FOLLOWER]: Refusing update from remote peer 46f1c841d0944835a4c09c4f9c0542a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:57:08.084002 24770 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 46f1c841d0944835a4c09c4f9c0542a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fd097b9153c4df890be21fc3a4282f1" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:57:08.084185 24770 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 46f1c841d0944835a4c09c4f9c0542a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddb9cd78ee4b4024b0066ad8ba658b14" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 42569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:57:08.085592 24757 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddb9cd78ee4b4024b0066ad8ba658b14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46f1c841d0944835a4c09c4f9c0542a4. Latest consensus state: current_term: 1 leader_uuid: "46f1c841d0944835a4c09c4f9c0542a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb9cd78ee4b4024b0066ad8ba658b14" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 42569 } } peers { permanent_uuid: "5fd097b9153c4df890be21fc3a4282f1" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45239 } } peers { permanent_uuid: "46f1c841d0944835a4c09c4f9c0542a4" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 39815 } } }
I20260929 10:57:08.085690 24757 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddb9cd78ee4b4024b0066ad8ba658b14 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:57:08.088791 24722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fd097b9153c4df890be21fc3a4282f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46f1c841d0944835a4c09c4f9c0542a4. Latest consensus state: current_term: 1 leader_uuid: "46f1c841d0944835a4c09c4f9c0542a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb9cd78ee4b4024b0066ad8ba658b14" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 42569 } } peers { permanent_uuid: "5fd097b9153c4df890be21fc3a4282f1" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45239 } } peers { permanent_uuid: "46f1c841d0944835a4c09c4f9c0542a4" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 39815 } } }
I20260929 10:57:08.088881 24722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fd097b9153c4df890be21fc3a4282f1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:57:08.090061 24770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46f1c841d0944835a4c09c4f9c0542a4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "46f1c841d0944835a4c09c4f9c0542a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb9cd78ee4b4024b0066ad8ba658b14" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 42569 } } peers { permanent_uuid: "5fd097b9153c4df890be21fc3a4282f1" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45239 } } peers { permanent_uuid: "46f1c841d0944835a4c09c4f9c0542a4" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 39815 } } }
I20260929 10:57:08.090155 24770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46f1c841d0944835a4c09c4f9c0542a4 [sys.catalog]: This master's current role is: LEADER
I20260929 10:57:08.090548 24757 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddb9cd78ee4b4024b0066ad8ba658b14 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "46f1c841d0944835a4c09c4f9c0542a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb9cd78ee4b4024b0066ad8ba658b14" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 42569 } } peers { permanent_uuid: "5fd097b9153c4df890be21fc3a4282f1" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45239 } } peers { permanent_uuid: "46f1c841d0944835a4c09c4f9c0542a4" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 39815 } } }
I20260929 10:57:08.090634 24757 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddb9cd78ee4b4024b0066ad8ba658b14 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:57:08.090821 24773 mvcc.cc:205] Tried to move back new op lower bound from 7334622937427795968 to 7334622937403379712. Current Snapshot: MvccSnapshot[applied={T|T < 7334622937427795968}]
I20260929 10:57:08.091341 24774 mvcc.cc:205] Tried to move back new op lower bound from 7334622937427795968 to 7334622937403379712. Current Snapshot: MvccSnapshot[applied={T|T < 7334622937427795968}]
I20260929 10:57:08.092373 24736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46f1c841d0944835a4c09c4f9c0542a4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "46f1c841d0944835a4c09c4f9c0542a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb9cd78ee4b4024b0066ad8ba658b14" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 42569 } } peers { permanent_uuid: "5fd097b9153c4df890be21fc3a4282f1" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45239 } } peers { permanent_uuid: "46f1c841d0944835a4c09c4f9c0542a4" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 39815 } } }
I20260929 10:57:08.092458 24736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46f1c841d0944835a4c09c4f9c0542a4 [sys.catalog]: This master's current role is: LEADER
I20260929 10:57:08.092832 24722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fd097b9153c4df890be21fc3a4282f1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "46f1c841d0944835a4c09c4f9c0542a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb9cd78ee4b4024b0066ad8ba658b14" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 42569 } } peers { permanent_uuid: "5fd097b9153c4df890be21fc3a4282f1" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 45239 } } peers { permanent_uuid: "46f1c841d0944835a4c09c4f9c0542a4" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 39815 } } }
I20260929 10:57:08.092918 24722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fd097b9153c4df890be21fc3a4282f1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:57:08.093133 24776 mvcc.cc:205] Tried to move back new op lower bound from 7334622937427795968 to 7334622937403379712. Current Snapshot: MvccSnapshot[applied={T|T < 7334622937427795968}]
I20260929 10:57:08.093667 24772 catalog_manager.cc:1347] Generated new cluster ID: 9f91788ac2a74724892d8207e113f739
I20260929 10:57:08.093730 24772 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:57:08.106859 24772 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:57:08.107599 24772 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:57:08.113436 24772 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46f1c841d0944835a4c09c4f9c0542a4: Generated new TSK 0
I20260929 10:57:08.113629 24772 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:57:09.018187 24733 catalog_manager.cc:1259] Loaded cluster ID: 9f91788ac2a74724892d8207e113f739
I20260929 10:57:09.018249 24733 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5fd097b9153c4df890be21fc3a4282f1: loading cluster ID for follower catalog manager: success
I20260929 10:57:09.019017 24733 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5fd097b9153c4df890be21fc3a4282f1: acquiring CA information for follower catalog manager: success
I20260929 10:57:09.019259 24733 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5fd097b9153c4df890be21fc3a4282f1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 10:57:09.064138 24768 catalog_manager.cc:1259] Loaded cluster ID: 9f91788ac2a74724892d8207e113f739
I20260929 10:57:09.064213 24768 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ddb9cd78ee4b4024b0066ad8ba658b14: loading cluster ID for follower catalog manager: success
I20260929 10:57:09.065052 24768 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ddb9cd78ee4b4024b0066ad8ba658b14: acquiring CA information for follower catalog manager: success
I20260929 10:57:09.065399 24768 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ddb9cd78ee4b4024b0066ad8ba658b14: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 10:57:09.173889 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 24496
I20260929 10:57:09.179427 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 24571
I20260929 10:57:09.184934 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 24646
2026-09-29T10:57:09Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2754 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790679274510869-11195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:57:09 dist-test-slave-8nrt krb5kdc[24788](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:57:09 dist-test-slave-8nrt krb5kdc[24788](info): set up 2 sockets
Sep 29 10:57:09 dist-test-slave-8nrt krb5kdc[24788](info): commencing operation
krb5kdc: starting...
W20260929 10:57:10.225365 11195 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Sep 29 10:57:10 dist-test-slave-8nrt krb5kdc[24788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:57:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:57:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
I20260929 10:57:10.343510 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:43229
--webserver_interface=127.10.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:37211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:43229
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:57:10.423878 24801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:57:10.424158 24801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:57:10.424201 24801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:57:10.425450 24801 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:57:10.425530 24801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:57:10.425555 24801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:57:10.425578 24801 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:57:10.425608 24801 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:57:10.427601 24801 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:37211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790679274510869-11195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:43229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:43229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:57:10.427896 24801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:57:10.428175 24801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:57:10.433791 24807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:10.433817 24809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:10.433808 24806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:57:10.434212 24801 server_base.cc:1034] running on GCE node
I20260929 10:57:10.434463 24801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:57:10.434715 24801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:57:10.435863 24801 hybrid_clock.cc:648] HybridClock initialized: now 1790679430435851 us; error 33 us; skew 500 ppm
Sep 29 10:57:10 dist-test-slave-8nrt krb5kdc[24788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679430, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:57:10.438614 24801 init.cc:377] Logged in from keytab as kudu/127.10.238.254@KRBTEST.COM (short username kudu)
I20260929 10:57:10.439137 24801 webserver.cc:460] Webserver started at http://127.10.238.254:35365/ using document root <none> and password file <none>
I20260929 10:57:10.439378 24801 fs_manager.cc:359] Metadata directory not provided
I20260929 10:57:10.439447 24801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:57:10.439610 24801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:57:10.440702 24801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790679274510869-11195-0/minicluster-data/master-0/data/instance:
uuid: "6c42ae1367f2497185cc1d44793cc83b"
format_stamp: "Formatted at 2026-09-29 10:57:10 on dist-test-slave-8nrt"
server_key: "7ea8ac12eea38e67ce0ea40be1bd4fdc"
server_key_iv: "b01973d9d64438728b931c7da2ac7202"
server_key_version: "encryptionkey@0"
I20260929 10:57:10.441088 24801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790679274510869-11195-0/minicluster-data/master-0/wal/instance:
uuid: "6c42ae1367f2497185cc1d44793cc83b"
format_stamp: "Formatted at 2026-09-29 10:57:10 on dist-test-slave-8nrt"
server_key: "7ea8ac12eea38e67ce0ea40be1bd4fdc"
server_key_iv: "b01973d9d64438728b931c7da2ac7202"
server_key_version: "encryptionkey@0"
I20260929 10:57:10.442603 24801 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 10:57:10.443405 24816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:10.443593 24801 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 10:57:10.443696 24801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "6c42ae1367f2497185cc1d44793cc83b"
format_stamp: "Formatted at 2026-09-29 10:57:10 on dist-test-slave-8nrt"
server_key: "7ea8ac12eea38e67ce0ea40be1bd4fdc"
server_key_iv: "b01973d9d64438728b931c7da2ac7202"
server_key_version: "encryptionkey@0"
I20260929 10:57:10.443785 24801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:57:10.458302 24801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:57:10.460831 24801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:57:10.461007 24801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:57:10.465165 24801 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.254:43229
I20260929 10:57:10.465219 24868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.254:43229 every 8 connection(s)
I20260929 10:57:10.465622 24801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:57:10.466512 24869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:57:10.468261 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 24801
I20260929 10:57:10.468355 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790679274510869-11195-0/minicluster-data/master-0/wal/instance
I20260929 10:57:10.468580 11195 external_mini_cluster.cc:1428] Setting key 54828638c489a44de4248e21cb9765f6
I20260929 10:57:10.469103 24869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c42ae1367f2497185cc1d44793cc83b: Bootstrap starting.
I20260929 10:57:10.469777 24869 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c42ae1367f2497185cc1d44793cc83b: Neither blocks nor log segments found. Creating new log.
I20260929 10:57:10.470108 24869 log.cc:824] T 00000000000000000000000000000000 P 6c42ae1367f2497185cc1d44793cc83b: Log is configured to *not* fsync() on all Append() calls
I20260929 10:57:10.470940 24869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c42ae1367f2497185cc1d44793cc83b: No bootstrap required, opened a new log
I20260929 10:57:10.472978 24869 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c42ae1367f2497185cc1d44793cc83b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c42ae1367f2497185cc1d44793cc83b" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 43229 } }
I20260929 10:57:10.473209 24869 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c42ae1367f2497185cc1d44793cc83b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:57:10.473318 24869 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c42ae1367f2497185cc1d44793cc83b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c42ae1367f2497185cc1d44793cc83b, State: Initialized, Role: FOLLOWER
I20260929 10:57:10.473477 24869 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c42ae1367f2497185cc1d44793cc83b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c42ae1367f2497185cc1d44793cc83b" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 43229 } }
I20260929 10:57:10.473567 24869 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c42ae1367f2497185cc1d44793cc83b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:57:10.473590 24869 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c42ae1367f2497185cc1d44793cc83b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:57:10.473660 24869 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c42ae1367f2497185cc1d44793cc83b [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:57:10.474284 24869 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c42ae1367f2497185cc1d44793cc83b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c42ae1367f2497185cc1d44793cc83b" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 43229 } }
I20260929 10:57:10.474404 24869 leader_election.cc:302] T 00000000000000000000000000000000 P 6c42ae1367f2497185cc1d44793cc83b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c42ae1367f2497185cc1d44793cc83b; no voters:
I20260929 10:57:10.474578 24869 leader_election.cc:288] T 00000000000000000000000000000000 P 6c42ae1367f2497185cc1d44793cc83b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:57:10.474630 24874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c42ae1367f2497185cc1d44793cc83b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:57:10.474826 24874 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c42ae1367f2497185cc1d44793cc83b [term 1 LEADER]: Becoming Leader. State: Replica: 6c42ae1367f2497185cc1d44793cc83b, State: Running, Role: LEADER
I20260929 10:57:10.474978 24874 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c42ae1367f2497185cc1d44793cc83b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c42ae1367f2497185cc1d44793cc83b" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 43229 } }
I20260929 10:57:10.474858 24869 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c42ae1367f2497185cc1d44793cc83b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:57:10.475368 24876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c42ae1367f2497185cc1d44793cc83b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c42ae1367f2497185cc1d44793cc83b. Latest consensus state: current_term: 1 leader_uuid: "6c42ae1367f2497185cc1d44793cc83b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c42ae1367f2497185cc1d44793cc83b" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 43229 } } }
I20260929 10:57:10.475368 24875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c42ae1367f2497185cc1d44793cc83b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c42ae1367f2497185cc1d44793cc83b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c42ae1367f2497185cc1d44793cc83b" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 43229 } } }
I20260929 10:57:10.475471 24876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c42ae1367f2497185cc1d44793cc83b [sys.catalog]: This master's current role is: LEADER
I20260929 10:57:10.475476 24875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c42ae1367f2497185cc1d44793cc83b [sys.catalog]: This master's current role is: LEADER
I20260929 10:57:10.475770 24878 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:57:10.476539 24878 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 29 10:57:10 dist-test-slave-8nrt krb5kdc[24788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:57:10.478390 24878 catalog_manager.cc:1347] Generated new cluster ID: 55708fa39f4f4230ac66a3dd922ac54b
I20260929 10:57:10.478446 24878 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:57:10.481122 24872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:10.469113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52928 (local address 127.10.238.254:43229)
0929 10:57:10.469328 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:10.469337 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:10.469363 (+ 26us) server_negotiation.cc:407] Connection header received
0929 10:57:10.469780 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:10.469787 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:10.469877 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:10.469997 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:57:10.470931 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:10.472280 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:10.473508 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:10.474042 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:10.474124 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:10.477139 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:57:10.477143 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:57:10.477143 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:57:10.477155 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:57:10.479215 (+ 2060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:10.479597 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:10.479600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:10.479601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:10.479689 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:10.479859 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:10.479860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:10.479860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:10.480123 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:57:10.480251 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:10.480728 (+ 477us) server_negotiation.cc:299] Negotiation successful
0929 10:57:10.480898 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":84,"threads_started":1}
I20260929 10:57:10.482646 11195 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:57:10.488864 24878 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:57:10.489444 24878 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:57:10.495723 24878 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c42ae1367f2497185cc1d44793cc83b: Generated new TSK 0
I20260929 10:57:10.495919 24878 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:57:10.536677 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 24801
2026-09-29T10:57:10Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1349 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:57:10 dist-test-slave-8nrt krb5kdc[24896](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:57:10 dist-test-slave-8nrt krb5kdc[24896](info): set up 2 sockets
Sep 29 10:57:10 dist-test-slave-8nrt krb5kdc[24896](info): commencing operation
krb5kdc: starting...
W20260929 10:57:11.569247 11195 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Sep 29 10:57:11 dist-test-slave-8nrt krb5kdc[24896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:57:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:57:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
I20260929 10:57:11.688915 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:40781
--webserver_interface=127.10.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:42761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:40781
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:57:11.770640 24909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:57:11.770860 24909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:57:11.770882 24909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:57:11.772058 24909 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:57:11.772104 24909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:57:11.772119 24909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:57:11.772132 24909 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:57:11.772188 24909 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:57:11.773823 24909 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:42761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:40781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:40781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:57:11.774094 24909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:57:11.774329 24909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:57:11.779333 24914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:57:11.779397 24909 server_base.cc:1034] running on GCE node
W20260929 10:57:11.779327 24917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:11.779327 24915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:57:11.779882 24909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:57:11.780180 24909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:57:11.781340 24909 hybrid_clock.cc:648] HybridClock initialized: now 1790679431781326 us; error 30 us; skew 500 ppm
Sep 29 10:57:11 dist-test-slave-8nrt krb5kdc[24896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679431, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:57:11.783895 24909 init.cc:377] Logged in from keytab as kudu/127.10.238.254@KRBTEST.COM (short username kudu)
I20260929 10:57:11.784324 24909 webserver.cc:460] Webserver started at http://127.10.238.254:40077/ using document root <none> and password file <none>
I20260929 10:57:11.784554 24909 fs_manager.cc:359] Metadata directory not provided
I20260929 10:57:11.784600 24909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:57:11.784765 24909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:57:11.785879 24909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/master-0/data/instance:
uuid: "98271e7b5d3848a7bd8941df8d73c321"
format_stamp: "Formatted at 2026-09-29 10:57:11 on dist-test-slave-8nrt"
server_key: "9f0450d2f78c5552cf3ed8c7e823e9de"
server_key_iv: "d0ca58d493c8b26db9b3e66341dcdc06"
server_key_version: "encryptionkey@0"
I20260929 10:57:11.786238 24909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/master-0/wal/instance:
uuid: "98271e7b5d3848a7bd8941df8d73c321"
format_stamp: "Formatted at 2026-09-29 10:57:11 on dist-test-slave-8nrt"
server_key: "9f0450d2f78c5552cf3ed8c7e823e9de"
server_key_iv: "d0ca58d493c8b26db9b3e66341dcdc06"
server_key_version: "encryptionkey@0"
I20260929 10:57:11.787516 24909 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 10:57:11.788215 24924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:11.788416 24909 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:57:11.788509 24909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "98271e7b5d3848a7bd8941df8d73c321"
format_stamp: "Formatted at 2026-09-29 10:57:11 on dist-test-slave-8nrt"
server_key: "9f0450d2f78c5552cf3ed8c7e823e9de"
server_key_iv: "d0ca58d493c8b26db9b3e66341dcdc06"
server_key_version: "encryptionkey@0"
I20260929 10:57:11.788584 24909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:57:11.796566 24909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:57:11.798935 24909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:57:11.799100 24909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:57:11.803020 24909 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.254:40781
I20260929 10:57:11.803092 24976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.254:40781 every 8 connection(s)
I20260929 10:57:11.803460 24909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:57:11.803952 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 24909
I20260929 10:57:11.804082 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/master-0/wal/instance
I20260929 10:57:11.804291 11195 external_mini_cluster.cc:1428] Setting key b52e7af8dda67f78e514f2edc209c3f4
I20260929 10:57:11.804618 24977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:57:11.808061 24977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321: Bootstrap starting.
I20260929 10:57:11.808732 24977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321: Neither blocks nor log segments found. Creating new log.
I20260929 10:57:11.809072 24977 log.cc:824] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321: Log is configured to *not* fsync() on all Append() calls
I20260929 10:57:11.809813 24977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321: No bootstrap required, opened a new log
Sep 29 10:57:11 dist-test-slave-8nrt krb5kdc[24896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:57:11.811754 24977 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98271e7b5d3848a7bd8941df8d73c321" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 40781 } }
I20260929 10:57:11.811910 24977 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:57:11.811959 24977 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98271e7b5d3848a7bd8941df8d73c321, State: Initialized, Role: FOLLOWER
I20260929 10:57:11.812111 24977 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98271e7b5d3848a7bd8941df8d73c321" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 40781 } }
I20260929 10:57:11.812234 24977 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:57:11.812271 24977 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:57:11.812331 24977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:57:11.812961 24977 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98271e7b5d3848a7bd8941df8d73c321" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 40781 } }
I20260929 10:57:11.813089 24977 leader_election.cc:302] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98271e7b5d3848a7bd8941df8d73c321; no voters:
I20260929 10:57:11.813301 24977 leader_election.cc:288] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:57:11.813313 24982 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:57:11.813467 24982 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321 [term 1 LEADER]: Becoming Leader. State: Replica: 98271e7b5d3848a7bd8941df8d73c321, State: Running, Role: LEADER
I20260929 10:57:11.813601 24977 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:57:11.813608 24982 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98271e7b5d3848a7bd8941df8d73c321" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 40781 } }
I20260929 10:57:11.814132 24984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98271e7b5d3848a7bd8941df8d73c321" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98271e7b5d3848a7bd8941df8d73c321" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 40781 } } }
I20260929 10:57:11.814216 24984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321 [sys.catalog]: This master's current role is: LEADER
I20260929 10:57:11.814448 24983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98271e7b5d3848a7bd8941df8d73c321. Latest consensus state: current_term: 1 leader_uuid: "98271e7b5d3848a7bd8941df8d73c321" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98271e7b5d3848a7bd8941df8d73c321" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 40781 } } }
I20260929 10:57:11.814777 24983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321 [sys.catalog]: This master's current role is: LEADER
I20260929 10:57:11.814960 24991 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:57:11.815650 24991 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:57:11.815831 24980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:11.804866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45788 (local address 127.10.238.254:40781)
0929 10:57:11.805071 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:11.805080 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:11.805115 (+ 35us) server_negotiation.cc:407] Connection header received
0929 10:57:11.805839 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:11.805848 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:11.805913 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:11.806073 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:57:11.806661 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:11.808029 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:11.808840 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:11.809448 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:11.809584 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:11.811902 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:57:11.811906 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:57:11.811906 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:57:11.811918 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:57:11.814038 (+ 2120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:11.814488 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:11.814491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:11.814491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:11.814593 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:11.814865 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:11.814866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:11.814866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:11.815198 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:57:11.815363 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:11.815525 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 10:57:11.815665 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":95,"threads_started":1}
I20260929 10:57:11.817191 24991 catalog_manager.cc:1347] Generated new cluster ID: 5af86ed08eff4ad587bf0a4e20da6be3
I20260929 10:57:11.817247 24991 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:57:11.823904 24991 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:57:11.824623 24991 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:57:11.836122 24991 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321: Generated new TSK 0
I20260929 10:57:11.836341 24991 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.238.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab.
Entry for principal kudu/127.10.238.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab.
I20260929 10:57:11.860602 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.193:0
--local_ip_for_outbound_sockets=127.10.238.193
--webserver_interface=127.10.238.193
--webserver_port=0
--tserver_master_addrs=127.10.238.254:40781
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab
--principal=kudu/127.10.238.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:42761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:57:11.942912 25003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:57:11.943151 25003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:57:11.943192 25003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:57:11.944356 25003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:57:11.944422 25003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:57:11.944468 25003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.193
I20260929 10:57:11.946027 25003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:42761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab
--principal=kudu/127.10.238.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.238.193
--webserver_port=0
--tserver_master_addrs=127.10.238.254:40781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.193
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:57:11.946310 25003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:57:11.946544 25003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:57:11.951812 25011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:11.951829 25009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:11.951812 25008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:57:11.952391 25003 server_base.cc:1034] running on GCE node
I20260929 10:57:11.952596 25003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:57:11.952872 25003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:57:11.954089 25003 hybrid_clock.cc:648] HybridClock initialized: now 1790679431954073 us; error 25 us; skew 500 ppm
Sep 29 10:57:11 dist-test-slave-8nrt krb5kdc[24896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679431, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:57:11.956730 25003 init.cc:377] Logged in from keytab as kudu/127.10.238.193@KRBTEST.COM (short username kudu)
I20260929 10:57:11.957114 25003 webserver.cc:460] Webserver started at http://127.10.238.193:40477/ using document root <none> and password file <none>
I20260929 10:57:11.957356 25003 fs_manager.cc:359] Metadata directory not provided
I20260929 10:57:11.957403 25003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:57:11.957491 25003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:57:11.958492 25003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-0/data/instance:
uuid: "7ef3ef9b6a7c449895a90ae689ebf8dc"
format_stamp: "Formatted at 2026-09-29 10:57:11 on dist-test-slave-8nrt"
server_key: "0f1b2a95b8a5ff390552d34e3929493d"
server_key_iv: "8f4287606c955b0856f2b0a758722d5f"
server_key_version: "encryptionkey@0"
I20260929 10:57:11.958796 25003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-0/wal/instance:
uuid: "7ef3ef9b6a7c449895a90ae689ebf8dc"
format_stamp: "Formatted at 2026-09-29 10:57:11 on dist-test-slave-8nrt"
server_key: "0f1b2a95b8a5ff390552d34e3929493d"
server_key_iv: "8f4287606c955b0856f2b0a758722d5f"
server_key_version: "encryptionkey@0"
I20260929 10:57:11.960052 25003 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 10:57:11.960769 25018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:11.960947 25003 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:57:11.961015 25003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-0/wal
uuid: "7ef3ef9b6a7c449895a90ae689ebf8dc"
format_stamp: "Formatted at 2026-09-29 10:57:11 on dist-test-slave-8nrt"
server_key: "0f1b2a95b8a5ff390552d34e3929493d"
server_key_iv: "8f4287606c955b0856f2b0a758722d5f"
server_key_version: "encryptionkey@0"
I20260929 10:57:11.961068 25003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:57:11.971118 25003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:57:11.972853 25003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:57:11.972961 25003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:57:11.973153 25003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:57:11.973475 25003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:57:11.973510 25003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:11.973531 25003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:57:11.973543 25003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:11.979589 25003 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.193:42879
I20260929 10:57:11.979666 25131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.193:42879 every 8 connection(s)
I20260929 10:57:11.980000 25003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
I20260929 10:57:11.985571 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 25003
I20260929 10:57:11.985664 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-0/wal/instance
I20260929 10:57:11.985896 11195 external_mini_cluster.cc:1428] Setting key 253100bf928fd5132f78f96413036317
Sep 29 10:57:11 dist-test-slave-8nrt krb5kdc[24896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679431, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.193@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:57:11.990366 24980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:11.980758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:41101 (local address 127.10.238.254:40781)
0929 10:57:11.980853 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:11.980856 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:11.981037 (+ 181us) server_negotiation.cc:407] Connection header received
0929 10:57:11.981672 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:11.981674 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:11.981685 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:11.981743 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:57:11.982707 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:11.983230 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:11.984246 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:11.984473 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:11.984560 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:11.987000 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:57:11.987001 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:57:11.987002 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:57:11.987010 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:57:11.988821 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:11.989232 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:11.989233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:11.989234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:11.989322 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:11.989618 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:11.989619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:11.989619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:11.989775 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:57:11.989853 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:11.990123 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 10:57:11.990227 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 10:57:11.990393 25134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:11.980732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.193:41101)
0929 10:57:11.980915 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:11.980951 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:11.981562 (+ 611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:11.981823 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:57:11.981827 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:57:11.982399 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:57:11.982590 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:11.982595 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:11.983335 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:11.983336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:11.984140 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:11.984142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:11.984573 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:11.984574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:11.984698 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:11.985249 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:57:11.985316 (+ 67us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:57:11.986902 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:57:11.988916 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:11.988924 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:11.988926 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:11.989155 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:11.989418 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:11.989419 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:11.989419 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:11.989538 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:11.989853 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:57:11.989859 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:57:11.989988 (+ 129us) client_negotiation.cc:769] Sending connection context
0929 10:57:11.990077 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 10:57:11.990221 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":86,"threads_started":1}
I20260929 10:57:11.991062 25132 heartbeater.cc:347] Connected to a master server at 127.10.238.254:40781
I20260929 10:57:11.991214 25132 heartbeater.cc:464] Registering TS with master...
I20260929 10:57:11.991451 25132 heartbeater.cc:511] Master 127.10.238.254:40781 requested a full tablet report, sending...
I20260929 10:57:11.992115 24941 ts_manager.cc:194] Registered new tserver with Master: 7ef3ef9b6a7c449895a90ae689ebf8dc (127.10.238.193:42879)
I20260929 10:57:11.993166 24941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.193@KRBTEST.COM'} at 127.10.238.193:41101
WARNING: no policy specified for kudu/127.10.238.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab.
Entry for principal kudu/127.10.238.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab.
I20260929 10:57:12.012771 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.194:0
--local_ip_for_outbound_sockets=127.10.238.194
--webserver_interface=127.10.238.194
--webserver_port=0
--tserver_master_addrs=127.10.238.254:40781
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab
--principal=kudu/127.10.238.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:42761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:57:12.095341 25137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:57:12.095551 25137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:57:12.095574 25137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:57:12.096705 25137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:57:12.096742 25137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:57:12.096767 25137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.194
I20260929 10:57:12.098172 25137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:42761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab
--principal=kudu/127.10.238.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.238.194
--webserver_port=0
--tserver_master_addrs=127.10.238.254:40781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.194
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:57:12.098353 25137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:57:12.098526 25137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 10:57:12.103832 25137 server_base.cc:1034] running on GCE node
W20260929 10:57:12.103782 25145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:12.103798 25142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:12.103773 25143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:57:12.104243 25137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:57:12.104524 25137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:57:12.105695 25137 hybrid_clock.cc:648] HybridClock initialized: now 1790679432105675 us; error 46 us; skew 500 ppm
Sep 29 10:57:12 dist-test-slave-8nrt krb5kdc[24896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679432, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:57:12.108337 25137 init.cc:377] Logged in from keytab as kudu/127.10.238.194@KRBTEST.COM (short username kudu)
I20260929 10:57:12.108764 25137 webserver.cc:460] Webserver started at http://127.10.238.194:44967/ using document root <none> and password file <none>
I20260929 10:57:12.108986 25137 fs_manager.cc:359] Metadata directory not provided
I20260929 10:57:12.109054 25137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:57:12.109205 25137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:57:12.110281 25137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-1/data/instance:
uuid: "bc5b041ccd1545fa949bed034f8066b8"
format_stamp: "Formatted at 2026-09-29 10:57:12 on dist-test-slave-8nrt"
server_key: "13fec2abed2246ad8a5095a75cf58d03"
server_key_iv: "71800f02a4a8bebedbbad49061ba5d8c"
server_key_version: "encryptionkey@0"
I20260929 10:57:12.110648 25137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-1/wal/instance:
uuid: "bc5b041ccd1545fa949bed034f8066b8"
format_stamp: "Formatted at 2026-09-29 10:57:12 on dist-test-slave-8nrt"
server_key: "13fec2abed2246ad8a5095a75cf58d03"
server_key_iv: "71800f02a4a8bebedbbad49061ba5d8c"
server_key_version: "encryptionkey@0"
I20260929 10:57:12.111985 25137 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 10:57:12.112715 25152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:12.112885 25137 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:57:12.112983 25137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-1/wal
uuid: "bc5b041ccd1545fa949bed034f8066b8"
format_stamp: "Formatted at 2026-09-29 10:57:12 on dist-test-slave-8nrt"
server_key: "13fec2abed2246ad8a5095a75cf58d03"
server_key_iv: "71800f02a4a8bebedbbad49061ba5d8c"
server_key_version: "encryptionkey@0"
I20260929 10:57:12.113066 25137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:57:12.134775 25137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:57:12.136719 25137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:57:12.136888 25137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:57:12.137140 25137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:57:12.137521 25137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:57:12.137574 25137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:12.137617 25137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:57:12.137642 25137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:12.143672 25137 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.194:40137
I20260929 10:57:12.143719 25265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.194:40137 every 8 connection(s)
I20260929 10:57:12.144033 25137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
I20260929 10:57:12.148110 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 25137
I20260929 10:57:12.148244 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-1/wal/instance
I20260929 10:57:12.148448 11195 external_mini_cluster.cc:1428] Setting key 39d4e881c7086c87a07abf8d76dfa729
Sep 29 10:57:12 dist-test-slave-8nrt krb5kdc[24896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679432, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.194@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:57:12.155169 24980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:12.144720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:54939 (local address 127.10.238.254:40781)
0929 10:57:12.144783 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:12.144785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:12.145004 (+ 219us) server_negotiation.cc:407] Connection header received
0929 10:57:12.145603 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:12.145604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:12.145615 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:12.145668 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:57:12.146622 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:12.147112 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:12.148187 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:12.148514 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:12.148593 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:12.152199 (+ 3606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:57:12.152201 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:57:12.152202 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:57:12.152210 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:57:12.153712 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:12.154139 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:12.154141 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:12.154141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:12.154203 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:12.154490 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:12.154491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:12.154491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:12.154611 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:57:12.154682 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:12.154978 (+ 296us) server_negotiation.cc:299] Negotiation successful
0929 10:57:12.155080 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 10:57:12.155376 25268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:12.144721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.194:54939)
0929 10:57:12.144896 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:12.144932 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:12.145507 (+ 575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:12.145772 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:57:12.145775 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:57:12.146347 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:57:12.146542 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:12.146547 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:12.147191 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:12.147192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:12.148094 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:12.148097 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:12.148627 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:12.148628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:12.148842 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:12.150596 (+ 1754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:57:12.150606 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:57:12.152096 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:57:12.153820 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:12.153827 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:12.153829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:12.154060 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:12.154271 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:12.154272 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:12.154273 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:12.154412 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:12.154685 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:57:12.154693 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:57:12.154838 (+ 145us) client_negotiation.cc:769] Sending connection context
0929 10:57:12.154944 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 10:57:12.155105 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":89,"threads_started":1}
I20260929 10:57:12.155813 25266 heartbeater.cc:347] Connected to a master server at 127.10.238.254:40781
I20260929 10:57:12.155897 25266 heartbeater.cc:464] Registering TS with master...
I20260929 10:57:12.156092 25266 heartbeater.cc:511] Master 127.10.238.254:40781 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.10.238.195@KRBTEST.COM; defaulting to no policy
I20260929 10:57:12.156528 24941 ts_manager.cc:194] Registered new tserver with Master: bc5b041ccd1545fa949bed034f8066b8 (127.10.238.194:40137)
I20260929 10:57:12.157019 24941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.194@KRBTEST.COM'} at 127.10.238.194:54939
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab.
Entry for principal kudu/127.10.238.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab.
I20260929 10:57:12.173892 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.195:0
--local_ip_for_outbound_sockets=127.10.238.195
--webserver_interface=127.10.238.195
--webserver_port=0
--tserver_master_addrs=127.10.238.254:40781
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab
--principal=kudu/127.10.238.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:42761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:57:12.256606 25271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:57:12.256814 25271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:57:12.256837 25271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:57:12.257933 25271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:57:12.257969 25271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:57:12.257995 25271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.195
I20260929 10:57:12.259344 25271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:42761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab
--principal=kudu/127.10.238.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.238.195
--webserver_port=0
--tserver_master_addrs=127.10.238.254:40781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.195
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:57:12.259522 25271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:57:12.259690 25271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:57:12.264662 25276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:12.264657 25279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:12.264667 25277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:57:12.264943 25271 server_base.cc:1034] running on GCE node
I20260929 10:57:12.265182 25271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:57:12.265455 25271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:57:12.266605 25271 hybrid_clock.cc:648] HybridClock initialized: now 1790679432266589 us; error 39 us; skew 500 ppm
Sep 29 10:57:12 dist-test-slave-8nrt krb5kdc[24896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679432, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:57:12.269145 25271 init.cc:377] Logged in from keytab as kudu/127.10.238.195@KRBTEST.COM (short username kudu)
I20260929 10:57:12.269610 25271 webserver.cc:460] Webserver started at http://127.10.238.195:41805/ using document root <none> and password file <none>
I20260929 10:57:12.269837 25271 fs_manager.cc:359] Metadata directory not provided
I20260929 10:57:12.269902 25271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:57:12.270048 25271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:57:12.271101 25271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-2/data/instance:
uuid: "c8210aa708674cd58c8cc8a70d1dcf68"
format_stamp: "Formatted at 2026-09-29 10:57:12 on dist-test-slave-8nrt"
server_key: "4953c7c4c36d2636e09b055834e005fb"
server_key_iv: "1dbddcd4b64f72137179b0316ce287b8"
server_key_version: "encryptionkey@0"
I20260929 10:57:12.271478 25271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-2/wal/instance:
uuid: "c8210aa708674cd58c8cc8a70d1dcf68"
format_stamp: "Formatted at 2026-09-29 10:57:12 on dist-test-slave-8nrt"
server_key: "4953c7c4c36d2636e09b055834e005fb"
server_key_iv: "1dbddcd4b64f72137179b0316ce287b8"
server_key_version: "encryptionkey@0"
I20260929 10:57:12.272763 25271 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 10:57:12.273515 25286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:12.273687 25271 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:12.273795 25271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-2/wal
uuid: "c8210aa708674cd58c8cc8a70d1dcf68"
format_stamp: "Formatted at 2026-09-29 10:57:12 on dist-test-slave-8nrt"
server_key: "4953c7c4c36d2636e09b055834e005fb"
server_key_iv: "1dbddcd4b64f72137179b0316ce287b8"
server_key_version: "encryptionkey@0"
I20260929 10:57:12.273870 25271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:57:12.289199 25271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:57:12.291700 25271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:57:12.291872 25271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:57:12.292120 25271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:57:12.292464 25271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:57:12.292514 25271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:12.292554 25271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:57:12.292584 25271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:12.298516 25271 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.195:38529
I20260929 10:57:12.298558 25399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.195:38529 every 8 connection(s)
I20260929 10:57:12.298920 25271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
Sep 29 10:57:12 dist-test-slave-8nrt krb5kdc[24896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679432, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.195@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:57:12.308595 24980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:12.299674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:49713 (local address 127.10.238.254:40781)
0929 10:57:12.299802 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:12.299805 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:12.299998 (+ 193us) server_negotiation.cc:407] Connection header received
0929 10:57:12.300570 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:12.300571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:12.300583 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:12.300624 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:57:12.301716 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:12.302232 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:12.303250 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:12.303463 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:12.303563 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:12.305760 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:57:12.305762 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:57:12.305762 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:57:12.305769 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:57:12.307186 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:12.307576 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:12.307578 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:12.307578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:12.307712 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:12.308030 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:12.308031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:12.308031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:12.308153 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:57:12.308219 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:12.308458 (+ 239us) server_negotiation.cc:299] Negotiation successful
0929 10:57:12.308529 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 10:57:12.308748 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 25271
I20260929 10:57:12.308859 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-2/wal/instance
I20260929 10:57:12.308856 25402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:12.299638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.195:49713)
0929 10:57:12.299855 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:12.299908 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:12.300452 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:12.300707 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:57:12.300711 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:57:12.301409 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:57:12.301609 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:12.301614 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:12.302333 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:12.302334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:12.303142 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:12.303144 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:12.303576 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:12.303577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:12.303703 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:12.304239 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:57:12.304248 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:57:12.305678 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:57:12.307259 (+ 1581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:12.307265 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:12.307266 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:12.307480 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:12.307782 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:12.307783 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:12.307783 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:12.307914 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:12.308221 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:57:12.308226 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:57:12.308344 (+ 118us) client_negotiation.cc:769] Sending connection context
0929 10:57:12.308425 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 10:57:12.308575 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":92,"threads_started":1}
I20260929 10:57:12.309083 11195 external_mini_cluster.cc:1428] Setting key 6379edeee9470c1ccab12f721eca2fd1
I20260929 10:57:12.309372 25400 heartbeater.cc:347] Connected to a master server at 127.10.238.254:40781
I20260929 10:57:12.309494 25400 heartbeater.cc:464] Registering TS with master...
I20260929 10:57:12.309752 25400 heartbeater.cc:511] Master 127.10.238.254:40781 requested a full tablet report, sending...
I20260929 10:57:12.310140 24941 ts_manager.cc:194] Registered new tserver with Master: c8210aa708674cd58c8cc8a70d1dcf68 (127.10.238.195:38529)
I20260929 10:57:12.310536 24941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.195@KRBTEST.COM'} at 127.10.238.195:49713
I20260929 10:57:12.321107 11195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:57:12.328485 24980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:12.322498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45804 (local address 127.10.238.254:40781)
0929 10:57:12.322599 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:12.322601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:12.322651 (+ 50us) server_negotiation.cc:407] Connection header received
0929 10:57:12.322690 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:12.322691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:12.322700 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:12.322756 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:57:12.323302 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:12.323966 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:12.324670 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:12.324863 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:12.324920 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:12.326220 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:57:12.326221 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:57:12.326221 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:57:12.326234 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:57:12.327445 (+ 1211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:12.327749 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:12.327749 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:12.327750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:12.327799 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:12.327958 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:12.327959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:12.327959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:12.328096 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:57:12.328176 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:12.328327 (+ 151us) server_negotiation.cc:299] Negotiation successful
0929 10:57:12.328388 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 10:57:12.330283 24941 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45804:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:57:12.331866 24941 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:57:12.338415 25410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:12.334162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.194:40137 (local address 127.0.0.1:56486)
0929 10:57:12.334576 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:12.334618 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:12.334698 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:12.334929 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:57:12.334931 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:57:12.334933 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:57:12.335005 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:12.335007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:12.335944 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:12.335946 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:12.337132 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:12.337134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:12.338075 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:12.338076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:12.338159 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:12.338173 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 10:57:12.338276 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 10:57:12.338335 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":76,"threads_started":1}
I20260929 10:57:12.338428 25411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:12.334368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56486 (local address 127.10.238.194:40137)
0929 10:57:12.334566 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:12.334571 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:12.334788 (+ 217us) server_negotiation.cc:407] Connection header received
0929 10:57:12.334838 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:12.334841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:12.334858 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:12.334931 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:57:12.335089 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:12.335836 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:12.337227 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:12.337982 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:12.338067 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:12.338128 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:12.338302 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 10:57:12.338345 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":108,"threads_started":1}
I20260929 10:57:12.338846 25409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:12.334171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.195:38529 (local address 127.0.0.1:45042)
0929 10:57:12.334640 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:12.334648 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:12.334698 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:12.335140 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:57:12.335141 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:57:12.335144 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:57:12.335180 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:12.335181 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:12.336528 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:12.336529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:12.337687 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:12.337690 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:12.338604 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:12.338605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:12.338670 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:12.338674 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 10:57:12.338726 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 10:57:12.338774 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":71,"threads_started":1}
I20260929 10:57:12.338927 25413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:12.334448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45042 (local address 127.10.238.195:38529)
0929 10:57:12.334650 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:12.334654 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:12.334691 (+ 37us) server_negotiation.cc:407] Connection header received
0929 10:57:12.334799 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:12.334802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:12.335029 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:12.335255 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:57:12.335345 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:12.336403 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:12.337791 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:12.338497 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:12.338593 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:12.338764 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:12.338792 (+ 28us) server_negotiation.cc:299] Negotiation successful
0929 10:57:12.338839 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":95,"threads_started":1}
I20260929 10:57:12.339402 25200 tablet_service.cc:1467] Processing CreateTablet for tablet d65a2bed215b4f5ba90eb49b5e9e5b13 (DEFAULT_TABLE table=test-table [id=95447019a80f4d1d9243f48a023c1ddd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:57:12.339609 25334 tablet_service.cc:1467] Processing CreateTablet for tablet d65a2bed215b4f5ba90eb49b5e9e5b13 (DEFAULT_TABLE table=test-table [id=95447019a80f4d1d9243f48a023c1ddd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:57:12.339823 25200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d65a2bed215b4f5ba90eb49b5e9e5b13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:57:12.339984 25334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d65a2bed215b4f5ba90eb49b5e9e5b13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:57:12.341430 25408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:12.334162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:42879 (local address 127.0.0.1:55846)
0929 10:57:12.334590 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:12.334776 (+ 186us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:12.334821 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:12.335071 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:57:12.335072 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:57:12.335075 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:57:12.335135 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:12.335136 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:12.336476 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:12.336477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:12.337497 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:12.337499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:12.338458 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:12.338459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:12.338743 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:12.338749 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 10:57:12.338802 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 10:57:12.338846 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":6,"thread_start_us":65,"threads_started":1}
I20260929 10:57:12.341557 25412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:12.334384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55846 (local address 127.10.238.193:42879)
0929 10:57:12.334656 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:12.334660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:12.334839 (+ 179us) server_negotiation.cc:407] Connection header received
0929 10:57:12.334884 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:12.334886 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:12.334901 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:12.334981 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:57:12.335215 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:12.336333 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:12.337596 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:12.338294 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:12.338369 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:12.338421 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:12.338890 (+ 469us) server_negotiation.cc:299] Negotiation successful
0929 10:57:12.339204 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":95,"threads_started":1}
I20260929 10:57:12.342410 25066 tablet_service.cc:1467] Processing CreateTablet for tablet d65a2bed215b4f5ba90eb49b5e9e5b13 (DEFAULT_TABLE table=test-table [id=95447019a80f4d1d9243f48a023c1ddd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:57:12.342718 25066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d65a2bed215b4f5ba90eb49b5e9e5b13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:57:12.342963 25414 tablet_bootstrap.cc:492] T d65a2bed215b4f5ba90eb49b5e9e5b13 P c8210aa708674cd58c8cc8a70d1dcf68: Bootstrap starting.
I20260929 10:57:12.343719 25414 tablet_bootstrap.cc:654] T d65a2bed215b4f5ba90eb49b5e9e5b13 P c8210aa708674cd58c8cc8a70d1dcf68: Neither blocks nor log segments found. Creating new log.
I20260929 10:57:12.344326 25414 log.cc:824] T d65a2bed215b4f5ba90eb49b5e9e5b13 P c8210aa708674cd58c8cc8a70d1dcf68: Log is configured to *not* fsync() on all Append() calls
I20260929 10:57:12.345129 25414 tablet_bootstrap.cc:492] T d65a2bed215b4f5ba90eb49b5e9e5b13 P c8210aa708674cd58c8cc8a70d1dcf68: No bootstrap required, opened a new log
I20260929 10:57:12.345211 25414 ts_tablet_manager.cc:1397] T d65a2bed215b4f5ba90eb49b5e9e5b13 P c8210aa708674cd58c8cc8a70d1dcf68: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 10:57:12.346259 25416 tablet_bootstrap.cc:492] T d65a2bed215b4f5ba90eb49b5e9e5b13 P bc5b041ccd1545fa949bed034f8066b8: Bootstrap starting.
I20260929 10:57:12.346885 25416 tablet_bootstrap.cc:654] T d65a2bed215b4f5ba90eb49b5e9e5b13 P bc5b041ccd1545fa949bed034f8066b8: Neither blocks nor log segments found. Creating new log.
I20260929 10:57:12.347270 25417 tablet_bootstrap.cc:492] T d65a2bed215b4f5ba90eb49b5e9e5b13 P 7ef3ef9b6a7c449895a90ae689ebf8dc: Bootstrap starting.
I20260929 10:57:12.347349 25414 raft_consensus.cc:348] T d65a2bed215b4f5ba90eb49b5e9e5b13 P c8210aa708674cd58c8cc8a70d1dcf68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc5b041ccd1545fa949bed034f8066b8" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40137 } } peers { permanent_uuid: "7ef3ef9b6a7c449895a90ae689ebf8dc" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 42879 } } peers { permanent_uuid: "c8210aa708674cd58c8cc8a70d1dcf68" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38529 } }
I20260929 10:57:12.347504 25416 log.cc:824] T d65a2bed215b4f5ba90eb49b5e9e5b13 P bc5b041ccd1545fa949bed034f8066b8: Log is configured to *not* fsync() on all Append() calls
I20260929 10:57:12.347545 25414 raft_consensus.cc:374] T d65a2bed215b4f5ba90eb49b5e9e5b13 P c8210aa708674cd58c8cc8a70d1dcf68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:57:12.347608 25414 raft_consensus.cc:729] T d65a2bed215b4f5ba90eb49b5e9e5b13 P c8210aa708674cd58c8cc8a70d1dcf68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8210aa708674cd58c8cc8a70d1dcf68, State: Initialized, Role: FOLLOWER
I20260929 10:57:12.347743 25414 consensus_queue.cc:260] T d65a2bed215b4f5ba90eb49b5e9e5b13 P c8210aa708674cd58c8cc8a70d1dcf68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc5b041ccd1545fa949bed034f8066b8" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40137 } } peers { permanent_uuid: "7ef3ef9b6a7c449895a90ae689ebf8dc" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 42879 } } peers { permanent_uuid: "c8210aa708674cd58c8cc8a70d1dcf68" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38529 } }
I20260929 10:57:12.348022 25417 tablet_bootstrap.cc:654] T d65a2bed215b4f5ba90eb49b5e9e5b13 P 7ef3ef9b6a7c449895a90ae689ebf8dc: Neither blocks nor log segments found. Creating new log.
I20260929 10:57:12.348116 25414 ts_tablet_manager.cc:1428] T d65a2bed215b4f5ba90eb49b5e9e5b13 P c8210aa708674cd58c8cc8a70d1dcf68: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260929 10:57:12.348127 25400 heartbeater.cc:503] Master 127.10.238.254:40781 was elected leader, sending a full tablet report...
I20260929 10:57:12.348381 25416 tablet_bootstrap.cc:492] T d65a2bed215b4f5ba90eb49b5e9e5b13 P bc5b041ccd1545fa949bed034f8066b8: No bootstrap required, opened a new log
I20260929 10:57:12.348461 25416 ts_tablet_manager.cc:1397] T d65a2bed215b4f5ba90eb49b5e9e5b13 P bc5b041ccd1545fa949bed034f8066b8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 10:57:12.348769 25417 log.cc:824] T d65a2bed215b4f5ba90eb49b5e9e5b13 P 7ef3ef9b6a7c449895a90ae689ebf8dc: Log is configured to *not* fsync() on all Append() calls
I20260929 10:57:12.349540 25417 tablet_bootstrap.cc:492] T d65a2bed215b4f5ba90eb49b5e9e5b13 P 7ef3ef9b6a7c449895a90ae689ebf8dc: No bootstrap required, opened a new log
I20260929 10:57:12.349627 25417 ts_tablet_manager.cc:1397] T d65a2bed215b4f5ba90eb49b5e9e5b13 P 7ef3ef9b6a7c449895a90ae689ebf8dc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 10:57:12.350284 25416 raft_consensus.cc:348] T d65a2bed215b4f5ba90eb49b5e9e5b13 P bc5b041ccd1545fa949bed034f8066b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc5b041ccd1545fa949bed034f8066b8" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40137 } } peers { permanent_uuid: "7ef3ef9b6a7c449895a90ae689ebf8dc" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 42879 } } peers { permanent_uuid: "c8210aa708674cd58c8cc8a70d1dcf68" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38529 } }
I20260929 10:57:12.350451 25416 raft_consensus.cc:374] T d65a2bed215b4f5ba90eb49b5e9e5b13 P bc5b041ccd1545fa949bed034f8066b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:57:12.350497 25416 raft_consensus.cc:729] T d65a2bed215b4f5ba90eb49b5e9e5b13 P bc5b041ccd1545fa949bed034f8066b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc5b041ccd1545fa949bed034f8066b8, State: Initialized, Role: FOLLOWER
I20260929 10:57:12.350615 25416 consensus_queue.cc:260] T d65a2bed215b4f5ba90eb49b5e9e5b13 P bc5b041ccd1545fa949bed034f8066b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc5b041ccd1545fa949bed034f8066b8" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40137 } } peers { permanent_uuid: "7ef3ef9b6a7c449895a90ae689ebf8dc" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 42879 } } peers { permanent_uuid: "c8210aa708674cd58c8cc8a70d1dcf68" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38529 } }
I20260929 10:57:12.350904 25416 ts_tablet_manager.cc:1428] T d65a2bed215b4f5ba90eb49b5e9e5b13 P bc5b041ccd1545fa949bed034f8066b8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 10:57:12.350929 25266 heartbeater.cc:503] Master 127.10.238.254:40781 was elected leader, sending a full tablet report...
I20260929 10:57:12.351344 25417 raft_consensus.cc:348] T d65a2bed215b4f5ba90eb49b5e9e5b13 P 7ef3ef9b6a7c449895a90ae689ebf8dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc5b041ccd1545fa949bed034f8066b8" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40137 } } peers { permanent_uuid: "7ef3ef9b6a7c449895a90ae689ebf8dc" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 42879 } } peers { permanent_uuid: "c8210aa708674cd58c8cc8a70d1dcf68" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38529 } }
I20260929 10:57:12.351497 25417 raft_consensus.cc:374] T d65a2bed215b4f5ba90eb49b5e9e5b13 P 7ef3ef9b6a7c449895a90ae689ebf8dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:57:12.351532 25417 raft_consensus.cc:729] T d65a2bed215b4f5ba90eb49b5e9e5b13 P 7ef3ef9b6a7c449895a90ae689ebf8dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ef3ef9b6a7c449895a90ae689ebf8dc, State: Initialized, Role: FOLLOWER
I20260929 10:57:12.351646 25417 consensus_queue.cc:260] T d65a2bed215b4f5ba90eb49b5e9e5b13 P 7ef3ef9b6a7c449895a90ae689ebf8dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc5b041ccd1545fa949bed034f8066b8" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40137 } } peers { permanent_uuid: "7ef3ef9b6a7c449895a90ae689ebf8dc" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 42879 } } peers { permanent_uuid: "c8210aa708674cd58c8cc8a70d1dcf68" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38529 } }
I20260929 10:57:12.351960 25417 ts_tablet_manager.cc:1428] T d65a2bed215b4f5ba90eb49b5e9e5b13 P 7ef3ef9b6a7c449895a90ae689ebf8dc: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 10:57:12.352160 25132 heartbeater.cc:503] Master 127.10.238.254:40781 was elected leader, sending a full tablet report...
W20260929 10:57:12.394593 25267 tablet.cc:2369] T d65a2bed215b4f5ba90eb49b5e9e5b13 P bc5b041ccd1545fa949bed034f8066b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 10:57:12.480798 25133 tablet.cc:2369] T d65a2bed215b4f5ba90eb49b5e9e5b13 P 7ef3ef9b6a7c449895a90ae689ebf8dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:57:12.514605 25419 raft_consensus.cc:482] T d65a2bed215b4f5ba90eb49b5e9e5b13 P c8210aa708674cd58c8cc8a70d1dcf68 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:57:12.514807 25419 raft_consensus.cc:504] T d65a2bed215b4f5ba90eb49b5e9e5b13 P c8210aa708674cd58c8cc8a70d1dcf68 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc5b041ccd1545fa949bed034f8066b8" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40137 } } peers { permanent_uuid: "7ef3ef9b6a7c449895a90ae689ebf8dc" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 42879 } } peers { permanent_uuid: "c8210aa708674cd58c8cc8a70d1dcf68" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38529 } }
I20260929 10:57:12.515231 25419 leader_election.cc:288] T d65a2bed215b4f5ba90eb49b5e9e5b13 P c8210aa708674cd58c8cc8a70d1dcf68 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc5b041ccd1545fa949bed034f8066b8 (127.10.238.194:40137), 7ef3ef9b6a7c449895a90ae689ebf8dc (127.10.238.193:42879)
I20260929 10:57:12.519318 25411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:12.515543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:41495 (local address 127.10.238.194:40137)
0929 10:57:12.515596 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:12.515598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:12.515767 (+ 169us) server_negotiation.cc:407] Connection header received
0929 10:57:12.515819 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:12.515819 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:12.515827 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:12.515866 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:57:12.516012 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:12.516776 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:12.518381 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:12.518956 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:12.519037 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:12.519046 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:12.519203 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 10:57:12.519243 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 10:57:12.519315 25412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:12.515519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:59133 (local address 127.10.238.193:42879)
0929 10:57:12.515613 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:12.515615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:12.515631 (+ 16us) server_negotiation.cc:407] Connection header received
0929 10:57:12.515658 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:12.515659 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:12.515667 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:12.515718 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:57:12.515966 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:12.516778 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:12.518381 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:12.518956 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:12.519037 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:12.519046 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:12.519203 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 10:57:12.519242 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 10:57:12.519340 25402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:12.515436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:42879 (local address 127.10.238.195:59133)
0929 10:57:12.515499 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:12.515511 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:12.515558 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:12.515772 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:57:12.515773 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:57:12.515775 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:57:12.515873 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:12.515876 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:12.516865 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:12.516866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:12.518280 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:12.518283 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:12.519046 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:12.519047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:12.519130 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:12.519132 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 10:57:12.519187 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 10:57:12.519241 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":71}
I20260929 10:57:12.519347 25423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:12.515468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.194:40137 (local address 127.10.238.195:41495)
0929 10:57:12.515673 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:12.515681 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:12.515756 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:12.515886 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:57:12.515887 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:57:12.515888 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:57:12.515951 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:12.515952 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:12.516865 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:12.516866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:12.518281 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:12.518283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:12.519046 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:12.519047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:12.519125 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:12.519129 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 10:57:12.519187 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 10:57:12.519248 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":113,"thread_start_us":90,"threads_started":1}
I20260929 10:57:12.519693 25086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d65a2bed215b4f5ba90eb49b5e9e5b13" candidate_uuid: "c8210aa708674cd58c8cc8a70d1dcf68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef3ef9b6a7c449895a90ae689ebf8dc" is_pre_election: true
I20260929 10:57:12.519830 25086 raft_consensus.cc:2454] T d65a2bed215b4f5ba90eb49b5e9e5b13 P 7ef3ef9b6a7c449895a90ae689ebf8dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8210aa708674cd58c8cc8a70d1dcf68 in term 0.
I20260929 10:57:12.520131 25288 leader_election.cc:302] T d65a2bed215b4f5ba90eb49b5e9e5b13 P c8210aa708674cd58c8cc8a70d1dcf68 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ef3ef9b6a7c449895a90ae689ebf8dc, c8210aa708674cd58c8cc8a70d1dcf68; no voters:
I20260929 10:57:12.520186 25220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d65a2bed215b4f5ba90eb49b5e9e5b13" candidate_uuid: "c8210aa708674cd58c8cc8a70d1dcf68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc5b041ccd1545fa949bed034f8066b8" is_pre_election: true
I20260929 10:57:12.520295 25419 raft_consensus.cc:2781] T d65a2bed215b4f5ba90eb49b5e9e5b13 P c8210aa708674cd58c8cc8a70d1dcf68 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:57:12.520370 25419 raft_consensus.cc:482] T d65a2bed215b4f5ba90eb49b5e9e5b13 P c8210aa708674cd58c8cc8a70d1dcf68 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:57:12.520390 25220 raft_consensus.cc:2454] T d65a2bed215b4f5ba90eb49b5e9e5b13 P bc5b041ccd1545fa949bed034f8066b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8210aa708674cd58c8cc8a70d1dcf68 in term 0.
I20260929 10:57:12.520406 25419 raft_consensus.cc:3037] T d65a2bed215b4f5ba90eb49b5e9e5b13 P c8210aa708674cd58c8cc8a70d1dcf68 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:57:12.521224 25419 raft_consensus.cc:504] T d65a2bed215b4f5ba90eb49b5e9e5b13 P c8210aa708674cd58c8cc8a70d1dcf68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc5b041ccd1545fa949bed034f8066b8" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40137 } } peers { permanent_uuid: "7ef3ef9b6a7c449895a90ae689ebf8dc" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 42879 } } peers { permanent_uuid: "c8210aa708674cd58c8cc8a70d1dcf68" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38529 } }
I20260929 10:57:12.521409 25419 leader_election.cc:288] T d65a2bed215b4f5ba90eb49b5e9e5b13 P c8210aa708674cd58c8cc8a70d1dcf68 [CANDIDATE]: Term 1 election: Requested vote from peers bc5b041ccd1545fa949bed034f8066b8 (127.10.238.194:40137), 7ef3ef9b6a7c449895a90ae689ebf8dc (127.10.238.193:42879)
I20260929 10:57:12.521605 25086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d65a2bed215b4f5ba90eb49b5e9e5b13" candidate_uuid: "c8210aa708674cd58c8cc8a70d1dcf68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef3ef9b6a7c449895a90ae689ebf8dc"
I20260929 10:57:12.521675 25086 raft_consensus.cc:3037] T d65a2bed215b4f5ba90eb49b5e9e5b13 P 7ef3ef9b6a7c449895a90ae689ebf8dc [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:57:12.522421 25086 raft_consensus.cc:2454] T d65a2bed215b4f5ba90eb49b5e9e5b13 P 7ef3ef9b6a7c449895a90ae689ebf8dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8210aa708674cd58c8cc8a70d1dcf68 in term 1.
I20260929 10:57:12.522562 25220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d65a2bed215b4f5ba90eb49b5e9e5b13" candidate_uuid: "c8210aa708674cd58c8cc8a70d1dcf68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc5b041ccd1545fa949bed034f8066b8"
I20260929 10:57:12.522595 25288 leader_election.cc:302] T d65a2bed215b4f5ba90eb49b5e9e5b13 P c8210aa708674cd58c8cc8a70d1dcf68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ef3ef9b6a7c449895a90ae689ebf8dc, c8210aa708674cd58c8cc8a70d1dcf68; no voters:
I20260929 10:57:12.522658 25220 raft_consensus.cc:3037] T d65a2bed215b4f5ba90eb49b5e9e5b13 P bc5b041ccd1545fa949bed034f8066b8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:57:12.522701 25419 raft_consensus.cc:2781] T d65a2bed215b4f5ba90eb49b5e9e5b13 P c8210aa708674cd58c8cc8a70d1dcf68 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:57:12.522848 25419 raft_consensus.cc:686] T d65a2bed215b4f5ba90eb49b5e9e5b13 P c8210aa708674cd58c8cc8a70d1dcf68 [term 1 LEADER]: Becoming Leader. State: Replica: c8210aa708674cd58c8cc8a70d1dcf68, State: Running, Role: LEADER
I20260929 10:57:12.522954 25419 consensus_queue.cc:237] T d65a2bed215b4f5ba90eb49b5e9e5b13 P c8210aa708674cd58c8cc8a70d1dcf68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc5b041ccd1545fa949bed034f8066b8" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40137 } } peers { permanent_uuid: "7ef3ef9b6a7c449895a90ae689ebf8dc" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 42879 } } peers { permanent_uuid: "c8210aa708674cd58c8cc8a70d1dcf68" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38529 } }
I20260929 10:57:12.523597 25220 raft_consensus.cc:2454] T d65a2bed215b4f5ba90eb49b5e9e5b13 P bc5b041ccd1545fa949bed034f8066b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8210aa708674cd58c8cc8a70d1dcf68 in term 1.
I20260929 10:57:12.523890 24941 catalog_manager.cc:5495] T d65a2bed215b4f5ba90eb49b5e9e5b13 P c8210aa708674cd58c8cc8a70d1dcf68 reported cstate change: term changed from 0 to 1, leader changed from <none> to c8210aa708674cd58c8cc8a70d1dcf68 (127.10.238.195). New cstate: current_term: 1 leader_uuid: "c8210aa708674cd58c8cc8a70d1dcf68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc5b041ccd1545fa949bed034f8066b8" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ef3ef9b6a7c449895a90ae689ebf8dc" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8210aa708674cd58c8cc8a70d1dcf68" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38529 } health_report { overall_health: HEALTHY } } }
W20260929 10:57:12.549610 25401 tablet.cc:2369] T d65a2bed215b4f5ba90eb49b5e9e5b13 P c8210aa708674cd58c8cc8a70d1dcf68: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:57:12.566885 25413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:12.564445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45054 (local address 127.10.238.195:38529)
0929 10:57:12.564510 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:12.564512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:12.564523 (+ 11us) server_negotiation.cc:407] Connection header received
0929 10:57:12.564563 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:12.564564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:12.564582 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:12.564643 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:57:12.564806 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:12.565341 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:12.566079 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:12.566277 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:12.566332 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:12.566522 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:57:12.566622 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:57:12.566683 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:12.566731 (+ 48us) server_negotiation.cc:299] Negotiation successful
0929 10:57:12.566751 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 10:57:12.568794 25086 raft_consensus.cc:1261] T d65a2bed215b4f5ba90eb49b5e9e5b13 P 7ef3ef9b6a7c449895a90ae689ebf8dc [term 1 FOLLOWER]: Refusing update from remote peer c8210aa708674cd58c8cc8a70d1dcf68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:57:12.568805 25220 raft_consensus.cc:1261] T d65a2bed215b4f5ba90eb49b5e9e5b13 P bc5b041ccd1545fa949bed034f8066b8 [term 1 FOLLOWER]: Refusing update from remote peer c8210aa708674cd58c8cc8a70d1dcf68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:57:12.569170 25424 consensus_queue.cc:1035] T d65a2bed215b4f5ba90eb49b5e9e5b13 P c8210aa708674cd58c8cc8a70d1dcf68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef3ef9b6a7c449895a90ae689ebf8dc" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:57:12.569343 25419 consensus_queue.cc:1035] T d65a2bed215b4f5ba90eb49b5e9e5b13 P c8210aa708674cd58c8cc8a70d1dcf68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc5b041ccd1545fa949bed034f8066b8" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:57:12.571813 25428 mvcc.cc:205] Tried to move back new op lower bound from 7334622955798876160 to 7334622955614621696. Current Snapshot: MvccSnapshot[applied={T|T < 7334622955798876160}]
I20260929 10:57:12.572150 25427 mvcc.cc:205] Tried to move back new op lower bound from 7334622955798876160 to 7334622955614621696. Current Snapshot: MvccSnapshot[applied={T|T < 7334622955798876160}]
I20260929 10:57:12.572281 25429 mvcc.cc:205] Tried to move back new op lower bound from 7334622955798876160 to 7334622955614621696. Current Snapshot: MvccSnapshot[applied={T|T < 7334622955798876160}]
I20260929 10:57:12.574434 24941 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:57:12.574535 24941 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:57:12.575273 24941 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321: Sending DeleteTablet for 3 replicas of tablet d65a2bed215b4f5ba90eb49b5e9e5b13
I20260929 10:57:12.575603 25200 tablet_service.cc:1514] Processing DeleteTablet for tablet d65a2bed215b4f5ba90eb49b5e9e5b13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:57:12 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:56486
I20260929 10:57:12.575642 25066 tablet_service.cc:1514] Processing DeleteTablet for tablet d65a2bed215b4f5ba90eb49b5e9e5b13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:57:12 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:55846
I20260929 10:57:12.575845 25334 tablet_service.cc:1514] Processing DeleteTablet for tablet d65a2bed215b4f5ba90eb49b5e9e5b13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:57:12 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:45042
I20260929 10:57:12.575947 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 25003
I20260929 10:57:12.576076 25439 tablet_replica.cc:330] T d65a2bed215b4f5ba90eb49b5e9e5b13 P c8210aa708674cd58c8cc8a70d1dcf68: stopping tablet replica
I20260929 10:57:12.576174 25439 raft_consensus.cc:2229] T d65a2bed215b4f5ba90eb49b5e9e5b13 P c8210aa708674cd58c8cc8a70d1dcf68 [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:57:12.577858 25439 raft_consensus.cc:2258] T d65a2bed215b4f5ba90eb49b5e9e5b13 P c8210aa708674cd58c8cc8a70d1dcf68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:57:12.578261 25439 ts_tablet_manager.cc:1905] T d65a2bed215b4f5ba90eb49b5e9e5b13 P c8210aa708674cd58c8cc8a70d1dcf68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:57:12.579967 25439 ts_tablet_manager.cc:1918] T d65a2bed215b4f5ba90eb49b5e9e5b13 P c8210aa708674cd58c8cc8a70d1dcf68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:57:12.580031 25439 log.cc:1196] T d65a2bed215b4f5ba90eb49b5e9e5b13 P c8210aa708674cd58c8cc8a70d1dcf68: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-2/wal/wals/d65a2bed215b4f5ba90eb49b5e9e5b13
I20260929 10:57:12.580344 25439 ts_tablet_manager.cc:1939] T d65a2bed215b4f5ba90eb49b5e9e5b13 P c8210aa708674cd58c8cc8a70d1dcf68: Deleting consensus metadata
I20260929 10:57:12.580778 24925 catalog_manager.cc:4841] TS c8210aa708674cd58c8cc8a70d1dcf68 (127.10.238.195:38529): tablet d65a2bed215b4f5ba90eb49b5e9e5b13 (table test-table [id=95447019a80f4d1d9243f48a023c1ddd]) successfully deleted
W20260929 10:57:12.581135 24926 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.238.193:42879 (error 108)
W20260929 10:57:12.581683 25408 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:57:12.581531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:42879 (local address 127.0.0.1:55858)
0929 10:57:12.581596 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:12.581637 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.193:42879: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 10:57:12.581809 24926 catalog_manager.cc:4584] TS 7ef3ef9b6a7c449895a90ae689ebf8dc (127.10.238.193:42879): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d65a2bed215b4f5ba90eb49b5e9e5b13: Network error: Client connection negotiation failed: client connection to 127.10.238.193:42879: connect: Connection reset by peer (error 104)
I20260929 10:57:12.581907 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 25137
I20260929 10:57:12.588011 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 25271
I20260929 10:57:12.588032 25408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:12.587929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.194:40137 (local address 127.0.0.1:56494)
0929 10:57:12.587976 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:12.587997 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.194:40137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260929 10:57:12.592876 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 24909
I20260929 10:57:12.598166 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:40781
--webserver_interface=127.10.238.254
--webserver_port=40077
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:42761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:40781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:57:12.679270 25441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:57:12.679478 25441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:57:12.679498 25441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:57:12.680574 25441 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:57:12.680608 25441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:57:12.680621 25441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:57:12.680634 25441 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:57:12.680646 25441 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:57:12.682047 25441 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:42761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:40781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:40781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.238.254
--webserver_port=40077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:57:12.682274 25441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:57:12.682500 25441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:57:12.693248 25448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:12.693302 25450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:12.693338 25447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:57:12.693467 25441 server_base.cc:1034] running on GCE node
I20260929 10:57:12.693823 25441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:57:12.694164 25441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:57:12.695348 25441 hybrid_clock.cc:648] HybridClock initialized: now 1790679432695328 us; error 49 us; skew 500 ppm
Sep 29 10:57:12 dist-test-slave-8nrt krb5kdc[24896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679432, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:57:12.701895 25441 init.cc:377] Logged in from keytab as kudu/127.10.238.254@KRBTEST.COM (short username kudu)
I20260929 10:57:12.702322 25441 webserver.cc:460] Webserver started at http://127.10.238.254:40077/ using document root <none> and password file <none>
I20260929 10:57:12.702519 25441 fs_manager.cc:359] Metadata directory not provided
I20260929 10:57:12.702569 25441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:57:12.704079 25441 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260929 10:57:12.704785 25457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:12.704965 25441 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:57:12.705050 25441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "98271e7b5d3848a7bd8941df8d73c321"
format_stamp: "Formatted at 2026-09-29 10:57:11 on dist-test-slave-8nrt"
server_key: "9f0450d2f78c5552cf3ed8c7e823e9de"
server_key_iv: "d0ca58d493c8b26db9b3e66341dcdc06"
server_key_version: "encryptionkey@0"
I20260929 10:57:12.705509 25441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:57:12.740446 25441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:57:12.743080 25441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:57:12.743202 25441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:57:12.747503 25441 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.254:40781
I20260929 10:57:12.747539 25509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.254:40781 every 8 connection(s)
I20260929 10:57:12.747956 25441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:57:12.749742 25510 sys_catalog.cc:263] Verifying existing consensus state
I20260929 10:57:12.750443 25510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321: Bootstrap starting.
I20260929 10:57:12.753508 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 25441
I20260929 10:57:12.753890 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.193:42879
--local_ip_for_outbound_sockets=127.10.238.193
--tserver_master_addrs=127.10.238.254:40781
--webserver_port=40477
--webserver_interface=127.10.238.193
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab
--principal=kudu/127.10.238.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:42761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:57:12.759732 25510 log.cc:824] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321: Log is configured to *not* fsync() on all Append() calls
I20260929 10:57:12.761766 25510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 10:57:12.762080 25510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321: Bootstrap complete.
I20260929 10:57:12.764402 25510 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98271e7b5d3848a7bd8941df8d73c321" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 40781 } }
I20260929 10:57:12.764647 25510 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98271e7b5d3848a7bd8941df8d73c321, State: Initialized, Role: FOLLOWER
I20260929 10:57:12.764858 25510 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98271e7b5d3848a7bd8941df8d73c321" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 40781 } }
I20260929 10:57:12.764955 25510 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:57:12.765013 25510 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:57:12.765060 25510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321 [term 1 FOLLOWER]: Advancing to term 2
I20260929 10:57:12.766137 25510 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98271e7b5d3848a7bd8941df8d73c321" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 40781 } }
I20260929 10:57:12.766264 25510 leader_election.cc:302] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98271e7b5d3848a7bd8941df8d73c321; no voters:
I20260929 10:57:12.766487 25510 leader_election.cc:288] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 10:57:12.766525 25514 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 10:57:12.766637 25514 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321 [term 2 LEADER]: Becoming Leader. State: Replica: 98271e7b5d3848a7bd8941df8d73c321, State: Running, Role: LEADER
I20260929 10:57:12.766769 25510 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:57:12.766815 25514 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98271e7b5d3848a7bd8941df8d73c321" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 40781 } }
I20260929 10:57:12.767251 25516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98271e7b5d3848a7bd8941df8d73c321. Latest consensus state: current_term: 2 leader_uuid: "98271e7b5d3848a7bd8941df8d73c321" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98271e7b5d3848a7bd8941df8d73c321" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 40781 } } }
I20260929 10:57:12.767256 25515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "98271e7b5d3848a7bd8941df8d73c321" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98271e7b5d3848a7bd8941df8d73c321" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 40781 } } }
I20260929 10:57:12.767359 25516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321 [sys.catalog]: This master's current role is: LEADER
I20260929 10:57:12.767374 25515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321 [sys.catalog]: This master's current role is: LEADER
I20260929 10:57:12.767765 25523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:57:12.768684 25523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:57:12.768970 25523 catalog_manager.cc:1259] Loaded cluster ID: 5af86ed08eff4ad587bf0a4e20da6be3
I20260929 10:57:12.769093 25523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:57:12.769394 25523 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 10:57:12.772123 25523 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 10:57:12.778932 25523 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:57:12.779170 25523 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321: Loaded TSK: 0
I20260929 10:57:12.779431 25523 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 10:57:12.866672 25512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:57:12.866955 25512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:57:12.866995 25512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:57:12.868680 25512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:57:12.868744 25512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:57:12.868786 25512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.193
I20260929 10:57:12.870909 25512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:42761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab
--principal=kudu/127.10.238.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.193:42879
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.238.193
--webserver_port=40477
--tserver_master_addrs=127.10.238.254:40781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.193
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:57:12.871148 25512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:57:12.871428 25512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:57:12.876904 25538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:12.876917 25540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:12.876917 25537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:57:12.877136 25512 server_base.cc:1034] running on GCE node
I20260929 10:57:12.877465 25512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:57:12.877746 25512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:57:12.878938 25512 hybrid_clock.cc:648] HybridClock initialized: now 1790679432878926 us; error 28 us; skew 500 ppm
Sep 29 10:57:12 dist-test-slave-8nrt krb5kdc[24896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679432, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:57:12.881914 25512 init.cc:377] Logged in from keytab as kudu/127.10.238.193@KRBTEST.COM (short username kudu)
I20260929 10:57:12.882349 25512 webserver.cc:460] Webserver started at http://127.10.238.193:40477/ using document root <none> and password file <none>
I20260929 10:57:12.882575 25512 fs_manager.cc:359] Metadata directory not provided
I20260929 10:57:12.882658 25512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:57:12.884070 25512 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:57:12.884827 25547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:12.885006 25512 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 10:57:12.885114 25512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-0/wal
uuid: "7ef3ef9b6a7c449895a90ae689ebf8dc"
format_stamp: "Formatted at 2026-09-29 10:57:11 on dist-test-slave-8nrt"
server_key: "0f1b2a95b8a5ff390552d34e3929493d"
server_key_iv: "8f4287606c955b0856f2b0a758722d5f"
server_key_version: "encryptionkey@0"
I20260929 10:57:12.885543 25512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:57:12.895602 25512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:57:12.898051 25512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:57:12.898227 25512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:57:12.898473 25512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:57:12.898933 25554 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 10:57:12.900951 25512 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 10:57:12.901022 25512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260929 10:57:12.901082 25512 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 10:57:12.901823 25512 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 10:57:12.901881 25512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260929 10:57:12.901947 25554 tablet_bootstrap.cc:492] T d65a2bed215b4f5ba90eb49b5e9e5b13 P 7ef3ef9b6a7c449895a90ae689ebf8dc: Bootstrap starting.
I20260929 10:57:12.910357 25512 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.193:42879
I20260929 10:57:12.910926 25512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
I20260929 10:57:12.913712 25554 log.cc:824] T d65a2bed215b4f5ba90eb49b5e9e5b13 P 7ef3ef9b6a7c449895a90ae689ebf8dc: Log is configured to *not* fsync() on all Append() calls
I20260929 10:57:12.913758 25661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.193:42879 every 8 connection(s)
I20260929 10:57:12.919822 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 25512
I20260929 10:57:12.920197 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.194:40137
--local_ip_for_outbound_sockets=127.10.238.194
--tserver_master_addrs=127.10.238.254:40781
--webserver_port=44967
--webserver_interface=127.10.238.194
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab
--principal=kudu/127.10.238.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:42761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:57:12.920528 25554 tablet_bootstrap.cc:492] T d65a2bed215b4f5ba90eb49b5e9e5b13 P 7ef3ef9b6a7c449895a90ae689ebf8dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 10:57:12.920853 25554 tablet_bootstrap.cc:492] T d65a2bed215b4f5ba90eb49b5e9e5b13 P 7ef3ef9b6a7c449895a90ae689ebf8dc: Bootstrap complete.
I20260929 10:57:12.922590 25554 ts_tablet_manager.cc:1397] T d65a2bed215b4f5ba90eb49b5e9e5b13 P 7ef3ef9b6a7c449895a90ae689ebf8dc: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.013s
Sep 29 10:57:12 dist-test-slave-8nrt krb5kdc[24896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679432, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.193@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:57:12.924422 25554 raft_consensus.cc:348] T d65a2bed215b4f5ba90eb49b5e9e5b13 P 7ef3ef9b6a7c449895a90ae689ebf8dc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc5b041ccd1545fa949bed034f8066b8" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40137 } } peers { permanent_uuid: "7ef3ef9b6a7c449895a90ae689ebf8dc" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 42879 } } peers { permanent_uuid: "c8210aa708674cd58c8cc8a70d1dcf68" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38529 } }
I20260929 10:57:12.924803 25554 raft_consensus.cc:729] T d65a2bed215b4f5ba90eb49b5e9e5b13 P 7ef3ef9b6a7c449895a90ae689ebf8dc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ef3ef9b6a7c449895a90ae689ebf8dc, State: Initialized, Role: FOLLOWER
I20260929 10:57:12.925354 25554 consensus_queue.cc:260] T d65a2bed215b4f5ba90eb49b5e9e5b13 P 7ef3ef9b6a7c449895a90ae689ebf8dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc5b041ccd1545fa949bed034f8066b8" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40137 } } peers { permanent_uuid: "7ef3ef9b6a7c449895a90ae689ebf8dc" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 42879 } } peers { permanent_uuid: "c8210aa708674cd58c8cc8a70d1dcf68" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38529 } }
I20260929 10:57:12.925689 25554 ts_tablet_manager.cc:1428] T d65a2bed215b4f5ba90eb49b5e9e5b13 P 7ef3ef9b6a7c449895a90ae689ebf8dc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 10:57:12.929147 25664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:12.911807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.193:57351)
0929 10:57:12.914047 (+ 2240us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:12.914082 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:12.914600 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:12.917885 (+ 3285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:57:12.917888 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:57:12.918186 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:57:12.918489 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:12.918496 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:12.920194 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:12.920195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:12.921479 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:12.921482 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:12.922312 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:12.922313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:12.922495 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:12.923109 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:57:12.923119 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:57:12.925022 (+ 1903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:57:12.927373 (+ 2351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:12.927380 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:12.927382 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:12.927614 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:12.927899 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:12.927900 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:12.927900 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:12.928017 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:12.928421 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:57:12.928428 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:57:12.928600 (+ 172us) client_negotiation.cc:769] Sending connection context
0929 10:57:12.928718 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 10:57:12.928927 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2198,"thread_start_us":114,"threads_started":1}
I20260929 10:57:12.929169 25665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:12.911898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:57351 (local address 127.10.238.254:40781)
0929 10:57:12.917421 (+ 5523us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:12.917431 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:12.917475 (+ 44us) server_negotiation.cc:407] Connection header received
0929 10:57:12.917611 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:12.917618 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:12.917678 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:12.917835 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:57:12.918638 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:12.920052 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:12.921580 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:12.922182 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:12.922285 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:12.925154 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:57:12.925158 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:57:12.925158 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:57:12.925171 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:57:12.927274 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:12.927713 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:12.927716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:12.927716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:12.927820 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:12.928093 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:12.928094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:12.928094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:12.928334 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:57:12.928429 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:12.928798 (+ 369us) server_negotiation.cc:299] Negotiation successful
0929 10:57:12.928932 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5468,"thread_start_us":75,"threads_started":1}
I20260929 10:57:12.930373 25662 heartbeater.cc:347] Connected to a master server at 127.10.238.254:40781
I20260929 10:57:12.930497 25662 heartbeater.cc:464] Registering TS with master...
I20260929 10:57:12.930698 25662 heartbeater.cc:511] Master 127.10.238.254:40781 requested a full tablet report, sending...
I20260929 10:57:12.931489 25474 ts_manager.cc:194] Registered new tserver with Master: 7ef3ef9b6a7c449895a90ae689ebf8dc (127.10.238.193:42879)
I20260929 10:57:12.932930 25474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.193@KRBTEST.COM'} at 127.10.238.193:57351
I20260929 10:57:12.934044 25662 heartbeater.cc:503] Master 127.10.238.254:40781 was elected leader, sending a full tablet report...
I20260929 10:57:12.944780 25669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:12.933422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:42879 (local address 127.0.0.1:55864)
0929 10:57:12.937450 (+ 4028us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:12.937483 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:12.937544 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:12.941587 (+ 4043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:57:12.941590 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:57:12.941593 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:57:12.941944 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:12.941947 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:12.942837 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:12.942838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:12.943702 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:12.943705 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:12.944422 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:12.944423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:12.944528 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:12.944538 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 10:57:12.944629 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 10:57:12.944688 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3996,"thread_start_us":117,"threads_started":1}
I20260929 10:57:12.944778 25670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:12.933695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55864 (local address 127.10.238.193:42879)
0929 10:57:12.941417 (+ 7722us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:12.941421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:12.941432 (+ 11us) server_negotiation.cc:407] Connection header received
0929 10:57:12.941464 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:12.941465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:12.941479 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:12.941581 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:57:12.942044 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:12.942741 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:12.943818 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:12.944320 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:12.944422 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:12.944481 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:12.944648 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 10:57:12.944695 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7680,"thread_start_us":73,"threads_started":1}
I20260929 10:57:12.945538 25578 tablet_service.cc:1514] Processing DeleteTablet for tablet d65a2bed215b4f5ba90eb49b5e9e5b13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:57:12 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:55864
I20260929 10:57:12.946161 25671 tablet_replica.cc:330] stopping tablet replica
I20260929 10:57:12.946285 25671 raft_consensus.cc:2229] T d65a2bed215b4f5ba90eb49b5e9e5b13 P 7ef3ef9b6a7c449895a90ae689ebf8dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:57:12.946374 25671 raft_consensus.cc:2258] T d65a2bed215b4f5ba90eb49b5e9e5b13 P 7ef3ef9b6a7c449895a90ae689ebf8dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:57:12.946707 25671 ts_tablet_manager.cc:1905] T d65a2bed215b4f5ba90eb49b5e9e5b13 P 7ef3ef9b6a7c449895a90ae689ebf8dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:57:12.948828 25671 ts_tablet_manager.cc:1918] T d65a2bed215b4f5ba90eb49b5e9e5b13 P 7ef3ef9b6a7c449895a90ae689ebf8dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:57:12.948922 25671 log.cc:1196] T d65a2bed215b4f5ba90eb49b5e9e5b13 P 7ef3ef9b6a7c449895a90ae689ebf8dc: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-0/wal/wals/d65a2bed215b4f5ba90eb49b5e9e5b13
I20260929 10:57:12.949172 25671 ts_tablet_manager.cc:1939] T d65a2bed215b4f5ba90eb49b5e9e5b13 P 7ef3ef9b6a7c449895a90ae689ebf8dc: Deleting consensus metadata
I20260929 10:57:12.949707 25459 catalog_manager.cc:4841] TS 7ef3ef9b6a7c449895a90ae689ebf8dc (127.10.238.193:42879): tablet d65a2bed215b4f5ba90eb49b5e9e5b13 (table test-table [id=95447019a80f4d1d9243f48a023c1ddd]) successfully deleted
W20260929 10:57:13.013475 25667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:57:13.013664 25667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:57:13.013687 25667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:57:13.014847 25667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:57:13.014889 25667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:57:13.014947 25667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.194
I20260929 10:57:13.016400 25667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:42761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab
--principal=kudu/127.10.238.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.194:40137
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.238.194
--webserver_port=44967
--tserver_master_addrs=127.10.238.254:40781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.194
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:57:13.016636 25667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:57:13.016860 25667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:57:13.021697 25676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:13.021667 25679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:13.021685 25677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:57:13.021853 25667 server_base.cc:1034] running on GCE node
I20260929 10:57:13.022193 25667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:57:13.022476 25667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:57:13.023650 25667 hybrid_clock.cc:648] HybridClock initialized: now 1790679433023639 us; error 30 us; skew 500 ppm
Sep 29 10:57:13 dist-test-slave-8nrt krb5kdc[24896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679433, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:57:13.026070 25667 init.cc:377] Logged in from keytab as kudu/127.10.238.194@KRBTEST.COM (short username kudu)
I20260929 10:57:13.026492 25667 webserver.cc:460] Webserver started at http://127.10.238.194:44967/ using document root <none> and password file <none>
I20260929 10:57:13.026702 25667 fs_manager.cc:359] Metadata directory not provided
I20260929 10:57:13.026757 25667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:57:13.028079 25667 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:57:13.028723 25686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:13.028905 25667 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:57:13.028991 25667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-1/wal
uuid: "bc5b041ccd1545fa949bed034f8066b8"
format_stamp: "Formatted at 2026-09-29 10:57:12 on dist-test-slave-8nrt"
server_key: "13fec2abed2246ad8a5095a75cf58d03"
server_key_iv: "71800f02a4a8bebedbbad49061ba5d8c"
server_key_version: "encryptionkey@0"
I20260929 10:57:13.029395 25667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:57:13.042052 25667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:57:13.043830 25667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:57:13.043982 25667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:57:13.044203 25667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:57:13.044634 25693 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 10:57:13.046029 25667 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 10:57:13.046090 25667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260929 10:57:13.046130 25667 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 10:57:13.046759 25667 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 10:57:13.046820 25667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260929 10:57:13.046868 25693 tablet_bootstrap.cc:492] T d65a2bed215b4f5ba90eb49b5e9e5b13 P bc5b041ccd1545fa949bed034f8066b8: Bootstrap starting.
I20260929 10:57:13.053824 25667 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.194:40137
I20260929 10:57:13.053932 25800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.194:40137 every 8 connection(s)
I20260929 10:57:13.054255 25667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
I20260929 10:57:13.057863 25693 log.cc:824] T d65a2bed215b4f5ba90eb49b5e9e5b13 P bc5b041ccd1545fa949bed034f8066b8: Log is configured to *not* fsync() on all Append() calls
I20260929 10:57:13.058977 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 25667
I20260929 10:57:13.059289 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.195:38529
--local_ip_for_outbound_sockets=127.10.238.195
--tserver_master_addrs=127.10.238.254:40781
--webserver_port=41805
--webserver_interface=127.10.238.195
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab
--principal=kudu/127.10.238.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:42761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:57:13.060603 25693 tablet_bootstrap.cc:492] T d65a2bed215b4f5ba90eb49b5e9e5b13 P bc5b041ccd1545fa949bed034f8066b8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 10:57:13.060940 25693 tablet_bootstrap.cc:492] T d65a2bed215b4f5ba90eb49b5e9e5b13 P bc5b041ccd1545fa949bed034f8066b8: Bootstrap complete.
I20260929 10:57:13.062367 25693 ts_tablet_manager.cc:1397] T d65a2bed215b4f5ba90eb49b5e9e5b13 P bc5b041ccd1545fa949bed034f8066b8: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.006s
I20260929 10:57:13.063802 25693 raft_consensus.cc:348] T d65a2bed215b4f5ba90eb49b5e9e5b13 P bc5b041ccd1545fa949bed034f8066b8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc5b041ccd1545fa949bed034f8066b8" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40137 } } peers { permanent_uuid: "7ef3ef9b6a7c449895a90ae689ebf8dc" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 42879 } } peers { permanent_uuid: "c8210aa708674cd58c8cc8a70d1dcf68" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38529 } }
I20260929 10:57:13.064072 25693 raft_consensus.cc:729] T d65a2bed215b4f5ba90eb49b5e9e5b13 P bc5b041ccd1545fa949bed034f8066b8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc5b041ccd1545fa949bed034f8066b8, State: Initialized, Role: FOLLOWER
Sep 29 10:57:13 dist-test-slave-8nrt krb5kdc[24896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679433, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.194@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:57:13.064239 25693 consensus_queue.cc:260] T d65a2bed215b4f5ba90eb49b5e9e5b13 P bc5b041ccd1545fa949bed034f8066b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc5b041ccd1545fa949bed034f8066b8" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40137 } } peers { permanent_uuid: "7ef3ef9b6a7c449895a90ae689ebf8dc" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 42879 } } peers { permanent_uuid: "c8210aa708674cd58c8cc8a70d1dcf68" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38529 } }
I20260929 10:57:13.064535 25693 ts_tablet_manager.cc:1428] T d65a2bed215b4f5ba90eb49b5e9e5b13 P bc5b041ccd1545fa949bed034f8066b8: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260929 10:57:13.068053 25665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:13.055015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:54239 (local address 127.10.238.254:40781)
0929 10:57:13.055082 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:13.055085 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:13.058269 (+ 3184us) server_negotiation.cc:407] Connection header received
0929 10:57:13.058805 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:13.058806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:13.058822 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:13.058911 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:57:13.059715 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:13.060214 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:13.061726 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:13.062018 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:13.062055 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:13.064724 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:57:13.064725 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:57:13.064726 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:57:13.064735 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:57:13.066504 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:13.067015 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:13.067016 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:13.067016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:13.067082 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:13.067331 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:13.067332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:13.067333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:13.067501 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:57:13.067560 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:13.067861 (+ 301us) server_negotiation.cc:299] Negotiation successful
0929 10:57:13.067958 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 10:57:13.068228 25803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:13.054916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.194:54239)
0929 10:57:13.058123 (+ 3207us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:13.058173 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:13.058688 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:13.058977 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:57:13.058981 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:57:13.059321 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:57:13.059617 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:13.059623 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:13.060322 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:13.060322 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:13.061607 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:13.061610 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:13.062134 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:13.062134 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:13.062238 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:13.062804 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:57:13.062814 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:57:13.064634 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:57:13.066599 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:13.066606 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:13.066607 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:13.066924 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:13.067149 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:13.067150 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:13.067150 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:13.067255 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:13.067564 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:57:13.067573 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:57:13.067697 (+ 124us) client_negotiation.cc:769] Sending connection context
0929 10:57:13.067849 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 10:57:13.068013 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3167,"thread_start_us":110,"threads_started":1}
I20260929 10:57:13.068632 25801 heartbeater.cc:347] Connected to a master server at 127.10.238.254:40781
I20260929 10:57:13.068742 25801 heartbeater.cc:464] Registering TS with master...
I20260929 10:57:13.068953 25801 heartbeater.cc:511] Master 127.10.238.254:40781 requested a full tablet report, sending...
I20260929 10:57:13.069578 25474 ts_manager.cc:194] Registered new tserver with Master: bc5b041ccd1545fa949bed034f8066b8 (127.10.238.194:40137)
I20260929 10:57:13.070168 25474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.194@KRBTEST.COM'} at 127.10.238.194:54239
I20260929 10:57:13.071063 25801 heartbeater.cc:503] Master 127.10.238.254:40781 was elected leader, sending a full tablet report...
Sep 29 10:57:13 dist-test-slave-8nrt krb5kdc[24896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679432, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for kudu/127.10.238.194@KRBTEST.COM
I20260929 10:57:13.077567 25669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:13.069906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.194:40137 (local address 127.0.0.1:56502)
0929 10:57:13.069958 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:13.069968 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:13.070000 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:13.070323 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:57:13.070324 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:57:13.070644 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:57:13.070677 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:13.070679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:13.071705 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:13.071706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:13.072196 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:13.072198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:13.072674 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:13.072674 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:13.072737 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:13.073238 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:57:13.073244 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:57:13.074708 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:57:13.076560 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:13.076567 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:13.076569 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:13.076755 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:13.076974 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:13.076975 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:13.076975 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:13.077027 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:13.077320 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:57:13.077334 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:57:13.077374 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 10:57:13.077428 (+ 54us) client_negotiation.cc:240] Negotiation successful
0929 10:57:13.077503 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260929 10:57:13.077670 25807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:13.069988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56502 (local address 127.10.238.194:40137)
0929 10:57:13.070134 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:13.070139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:13.070151 (+ 12us) server_negotiation.cc:407] Connection header received
0929 10:57:13.070185 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:13.070188 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:13.070203 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:13.070320 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:57:13.070771 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:13.071623 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:13.072293 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:13.072595 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:13.072675 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:13.074792 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:57:13.074800 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:57:13.074800 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:57:13.074807 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:57:13.076479 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:13.076840 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:13.076843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:13.076843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:13.076910 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:13.077084 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:13.077085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:13.077085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:13.077245 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:57:13.077340 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:13.077482 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 10:57:13.077579 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":70,"threads_started":1}
I20260929 10:57:13.077963 25735 tablet_service.cc:1514] Processing DeleteTablet for tablet d65a2bed215b4f5ba90eb49b5e9e5b13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:57:12 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:56502
I20260929 10:57:13.078604 25808 tablet_replica.cc:330] stopping tablet replica
I20260929 10:57:13.078735 25808 raft_consensus.cc:2229] T d65a2bed215b4f5ba90eb49b5e9e5b13 P bc5b041ccd1545fa949bed034f8066b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:57:13.078833 25808 raft_consensus.cc:2258] T d65a2bed215b4f5ba90eb49b5e9e5b13 P bc5b041ccd1545fa949bed034f8066b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:57:13.079142 25808 ts_tablet_manager.cc:1905] T d65a2bed215b4f5ba90eb49b5e9e5b13 P bc5b041ccd1545fa949bed034f8066b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:57:13.080965 25808 ts_tablet_manager.cc:1918] T d65a2bed215b4f5ba90eb49b5e9e5b13 P bc5b041ccd1545fa949bed034f8066b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:57:13.081058 25808 log.cc:1196] T d65a2bed215b4f5ba90eb49b5e9e5b13 P bc5b041ccd1545fa949bed034f8066b8: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-1/wal/wals/d65a2bed215b4f5ba90eb49b5e9e5b13
I20260929 10:57:13.081348 25808 ts_tablet_manager.cc:1939] T d65a2bed215b4f5ba90eb49b5e9e5b13 P bc5b041ccd1545fa949bed034f8066b8: Deleting consensus metadata
I20260929 10:57:13.081836 25460 catalog_manager.cc:4841] TS bc5b041ccd1545fa949bed034f8066b8 (127.10.238.194:40137): tablet d65a2bed215b4f5ba90eb49b5e9e5b13 (table test-table [id=95447019a80f4d1d9243f48a023c1ddd]) successfully deleted
W20260929 10:57:13.145052 25805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:57:13.145301 25805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:57:13.145363 25805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:57:13.146497 25805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:57:13.146562 25805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:57:13.146606 25805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.195
I20260929 10:57:13.148595 25805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:42761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab
--principal=kudu/127.10.238.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.195:38529
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.238.195
--webserver_port=41805
--tserver_master_addrs=127.10.238.254:40781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.195
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:57:13.148851 25805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:57:13.149101 25805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:57:13.154488 25816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:57:13.154471 25805 server_base.cc:1034] running on GCE node
W20260929 10:57:13.154484 25813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:13.154464 25814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:57:13.154933 25805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:57:13.155162 25805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:57:13.156345 25805 hybrid_clock.cc:648] HybridClock initialized: now 1790679433156329 us; error 45 us; skew 500 ppm
Sep 29 10:57:13 dist-test-slave-8nrt krb5kdc[24896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679433, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:57:13.158823 25805 init.cc:377] Logged in from keytab as kudu/127.10.238.195@KRBTEST.COM (short username kudu)
I20260929 10:57:13.159260 25805 webserver.cc:460] Webserver started at http://127.10.238.195:41805/ using document root <none> and password file <none>
I20260929 10:57:13.159500 25805 fs_manager.cc:359] Metadata directory not provided
I20260929 10:57:13.159585 25805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:57:13.160919 25805 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:57:13.161589 25823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:13.161761 25805 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 10:57:13.161870 25805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-2/wal
uuid: "c8210aa708674cd58c8cc8a70d1dcf68"
format_stamp: "Formatted at 2026-09-29 10:57:12 on dist-test-slave-8nrt"
server_key: "4953c7c4c36d2636e09b055834e005fb"
server_key_iv: "1dbddcd4b64f72137179b0316ce287b8"
server_key_version: "encryptionkey@0"
I20260929 10:57:13.162250 25805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:57:13.186779 25805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:57:13.188840 25805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:57:13.189023 25805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:57:13.189364 25805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:57:13.189718 25805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:57:13.189762 25805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:13.189817 25805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:57:13.189846 25805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:13.195642 25805 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.195:38529
I20260929 10:57:13.195744 25936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.195:38529 every 8 connection(s)
I20260929 10:57:13.196091 25805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
Sep 29 10:57:13 dist-test-slave-8nrt krb5kdc[24896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679433, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.195@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:57:13.205466 25665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:13.196931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:47443 (local address 127.10.238.254:40781)
0929 10:57:13.197001 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:13.197003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:13.197218 (+ 215us) server_negotiation.cc:407] Connection header received
0929 10:57:13.197805 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:13.197806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:13.197816 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:13.197879 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:57:13.198815 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:13.199300 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:13.200292 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:13.200483 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:13.200534 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:13.202651 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:57:13.202652 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:57:13.202653 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:57:13.202659 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:57:13.204104 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:13.204456 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:13.204456 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:13.204457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:13.204507 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:13.204753 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:13.204755 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:13.204755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:13.204890 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:57:13.204973 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:13.205243 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 10:57:13.205371 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 10:57:13.205634 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 25805
I20260929 10:57:13.205668 25939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:13.196924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.195:47443)
0929 10:57:13.197076 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:13.197120 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:13.197708 (+ 588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:13.197938 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:57:13.197941 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:57:13.198524 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:57:13.198730 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:13.198735 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:13.199397 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:13.199398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:13.200204 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:13.200206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:13.200559 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:13.200559 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:13.200658 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:13.201143 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:57:13.201151 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:57:13.202564 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:57:13.204174 (+ 1610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:13.204180 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:13.204181 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:13.204392 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:13.204566 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:13.204566 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:13.204566 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:13.204665 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:13.204964 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:57:13.204972 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:57:13.205107 (+ 135us) client_negotiation.cc:769] Sending connection context
0929 10:57:13.205220 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 10:57:13.205422 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":77,"threads_started":1}
I20260929 10:57:13.206090 25937 heartbeater.cc:347] Connected to a master server at 127.10.238.254:40781
I20260929 10:57:13.206221 25937 heartbeater.cc:464] Registering TS with master...
I20260929 10:57:13.206420 25937 heartbeater.cc:511] Master 127.10.238.254:40781 requested a full tablet report, sending...
I20260929 10:57:13.206826 25474 ts_manager.cc:194] Registered new tserver with Master: c8210aa708674cd58c8cc8a70d1dcf68 (127.10.238.195:38529)
I20260929 10:57:13.207314 25474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.195@KRBTEST.COM'} at 127.10.238.195:47443
I20260929 10:57:13.211944 25665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:13.206029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45818 (local address 127.10.238.254:40781)
0929 10:57:13.206066 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:13.206068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:13.206216 (+ 148us) server_negotiation.cc:407] Connection header received
0929 10:57:13.206283 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:13.206284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:13.206294 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:13.206343 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:57:13.206953 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:13.207618 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:13.208491 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:13.208734 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:13.208793 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:13.210004 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:57:13.210005 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:57:13.210005 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:57:13.210011 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:57:13.211110 (+ 1099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:13.211411 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:13.211412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:13.211412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:13.211453 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:13.211602 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:13.211603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:13.211603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:13.211685 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:57:13.211809 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:13.211835 (+ 26us) server_negotiation.cc:299] Negotiation successful
0929 10:57:13.211887 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 10:57:13.212644 11195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:57:13.215415 25665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:13.212982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45832 (local address 127.10.238.254:40781)
0929 10:57:13.213043 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:13.213045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:13.213113 (+ 68us) server_negotiation.cc:407] Connection header received
0929 10:57:13.213191 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:13.213191 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:13.213249 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:13.213344 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:57:13.213462 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:13.213908 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:13.214663 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:13.214859 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:13.214933 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:13.215104 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:57:13.215199 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:57:13.215305 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:13.215328 (+ 23us) server_negotiation.cc:299] Negotiation successful
0929 10:57:13.215345 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 10:57:13.215811 25474 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:45832:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:57:13.217497 25474 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:57:13.219561 25578 tablet_service.cc:1467] Processing CreateTablet for tablet eba7bac139fb4fafa1743db1f6661d09 (DEFAULT_TABLE table=test-table [id=567c3a823ced44b18d73aa912e9ba6a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:57:13.219688 25735 tablet_service.cc:1467] Processing CreateTablet for tablet eba7bac139fb4fafa1743db1f6661d09 (DEFAULT_TABLE table=test-table [id=567c3a823ced44b18d73aa912e9ba6a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:57:13.219761 25578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eba7bac139fb4fafa1743db1f6661d09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:57:13.219875 25735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eba7bac139fb4fafa1743db1f6661d09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:57:13.221248 25554 tablet_bootstrap.cc:492] T eba7bac139fb4fafa1743db1f6661d09 P 7ef3ef9b6a7c449895a90ae689ebf8dc: Bootstrap starting.
I20260929 10:57:13.221372 25693 tablet_bootstrap.cc:492] T eba7bac139fb4fafa1743db1f6661d09 P bc5b041ccd1545fa949bed034f8066b8: Bootstrap starting.
I20260929 10:57:13.221726 25554 tablet_bootstrap.cc:654] T eba7bac139fb4fafa1743db1f6661d09 P 7ef3ef9b6a7c449895a90ae689ebf8dc: Neither blocks nor log segments found. Creating new log.
I20260929 10:57:13.221750 25693 tablet_bootstrap.cc:654] T eba7bac139fb4fafa1743db1f6661d09 P bc5b041ccd1545fa949bed034f8066b8: Neither blocks nor log segments found. Creating new log.
I20260929 10:57:13.222234 25693 tablet_bootstrap.cc:492] T eba7bac139fb4fafa1743db1f6661d09 P bc5b041ccd1545fa949bed034f8066b8: No bootstrap required, opened a new log
I20260929 10:57:13.222285 25693 ts_tablet_manager.cc:1397] T eba7bac139fb4fafa1743db1f6661d09 P bc5b041ccd1545fa949bed034f8066b8: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260929 10:57:13.222328 25554 tablet_bootstrap.cc:492] T eba7bac139fb4fafa1743db1f6661d09 P 7ef3ef9b6a7c449895a90ae689ebf8dc: No bootstrap required, opened a new log
I20260929 10:57:13.222393 25554 ts_tablet_manager.cc:1397] T eba7bac139fb4fafa1743db1f6661d09 P 7ef3ef9b6a7c449895a90ae689ebf8dc: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 10:57:13.222554 25554 raft_consensus.cc:348] T eba7bac139fb4fafa1743db1f6661d09 P 7ef3ef9b6a7c449895a90ae689ebf8dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8210aa708674cd58c8cc8a70d1dcf68" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38529 } } peers { permanent_uuid: "7ef3ef9b6a7c449895a90ae689ebf8dc" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 42879 } } peers { permanent_uuid: "bc5b041ccd1545fa949bed034f8066b8" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40137 } }
I20260929 10:57:13.222645 25554 raft_consensus.cc:374] T eba7bac139fb4fafa1743db1f6661d09 P 7ef3ef9b6a7c449895a90ae689ebf8dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:57:13.222693 25554 raft_consensus.cc:729] T eba7bac139fb4fafa1743db1f6661d09 P 7ef3ef9b6a7c449895a90ae689ebf8dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ef3ef9b6a7c449895a90ae689ebf8dc, State: Initialized, Role: FOLLOWER
I20260929 10:57:13.222656 25693 raft_consensus.cc:348] T eba7bac139fb4fafa1743db1f6661d09 P bc5b041ccd1545fa949bed034f8066b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8210aa708674cd58c8cc8a70d1dcf68" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38529 } } peers { permanent_uuid: "7ef3ef9b6a7c449895a90ae689ebf8dc" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 42879 } } peers { permanent_uuid: "bc5b041ccd1545fa949bed034f8066b8" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40137 } }
I20260929 10:57:13.222718 25693 raft_consensus.cc:374] T eba7bac139fb4fafa1743db1f6661d09 P bc5b041ccd1545fa949bed034f8066b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:57:13.222775 25693 raft_consensus.cc:729] T eba7bac139fb4fafa1743db1f6661d09 P bc5b041ccd1545fa949bed034f8066b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc5b041ccd1545fa949bed034f8066b8, State: Initialized, Role: FOLLOWER
I20260929 10:57:13.222756 25554 consensus_queue.cc:260] T eba7bac139fb4fafa1743db1f6661d09 P 7ef3ef9b6a7c449895a90ae689ebf8dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8210aa708674cd58c8cc8a70d1dcf68" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38529 } } peers { permanent_uuid: "7ef3ef9b6a7c449895a90ae689ebf8dc" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 42879 } } peers { permanent_uuid: "bc5b041ccd1545fa949bed034f8066b8" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40137 } }
I20260929 10:57:13.222823 25693 consensus_queue.cc:260] T eba7bac139fb4fafa1743db1f6661d09 P bc5b041ccd1545fa949bed034f8066b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8210aa708674cd58c8cc8a70d1dcf68" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38529 } } peers { permanent_uuid: "7ef3ef9b6a7c449895a90ae689ebf8dc" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 42879 } } peers { permanent_uuid: "bc5b041ccd1545fa949bed034f8066b8" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40137 } }
I20260929 10:57:13.222857 25554 ts_tablet_manager.cc:1428] T eba7bac139fb4fafa1743db1f6661d09 P 7ef3ef9b6a7c449895a90ae689ebf8dc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:13.222908 25693 ts_tablet_manager.cc:1428] T eba7bac139fb4fafa1743db1f6661d09 P bc5b041ccd1545fa949bed034f8066b8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 10:57:13.225390 25669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:13.219345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.195:38529 (local address 127.0.0.1:45056)
0929 10:57:13.219454 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:13.219464 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:13.219503 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:13.222332 (+ 2829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:57:13.222333 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:57:13.222335 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:57:13.222377 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:13.222379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:13.223762 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:13.223763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:13.224467 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:13.224469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:13.225138 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:13.225138 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:13.225204 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:13.225207 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 10:57:13.225285 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 10:57:13.225328 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260929 10:57:13.225463 25942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:13.221362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45056 (local address 127.10.238.195:38529)
0929 10:57:13.222143 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:13.222148 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:13.222159 (+ 11us) server_negotiation.cc:407] Connection header received
0929 10:57:13.222202 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:13.222204 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:13.222217 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:13.222320 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:57:13.222455 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:13.223672 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:13.224564 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:13.225041 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:13.225129 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:13.225183 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:13.225321 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 10:57:13.225369 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":742,"thread_start_us":98,"threads_started":1}
I20260929 10:57:13.226188 25871 tablet_service.cc:1467] Processing CreateTablet for tablet eba7bac139fb4fafa1743db1f6661d09 (DEFAULT_TABLE table=test-table [id=567c3a823ced44b18d73aa912e9ba6a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:57:13.226518 25871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eba7bac139fb4fafa1743db1f6661d09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:57:13.229070 25945 tablet_bootstrap.cc:492] T eba7bac139fb4fafa1743db1f6661d09 P c8210aa708674cd58c8cc8a70d1dcf68: Bootstrap starting.
I20260929 10:57:13.229705 25945 tablet_bootstrap.cc:654] T eba7bac139fb4fafa1743db1f6661d09 P c8210aa708674cd58c8cc8a70d1dcf68: Neither blocks nor log segments found. Creating new log.
I20260929 10:57:13.230167 25945 log.cc:824] T eba7bac139fb4fafa1743db1f6661d09 P c8210aa708674cd58c8cc8a70d1dcf68: Log is configured to *not* fsync() on all Append() calls
I20260929 10:57:13.230836 25945 tablet_bootstrap.cc:492] T eba7bac139fb4fafa1743db1f6661d09 P c8210aa708674cd58c8cc8a70d1dcf68: No bootstrap required, opened a new log
I20260929 10:57:13.230891 25945 ts_tablet_manager.cc:1397] T eba7bac139fb4fafa1743db1f6661d09 P c8210aa708674cd58c8cc8a70d1dcf68: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 10:57:13.232455 25945 raft_consensus.cc:348] T eba7bac139fb4fafa1743db1f6661d09 P c8210aa708674cd58c8cc8a70d1dcf68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8210aa708674cd58c8cc8a70d1dcf68" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38529 } } peers { permanent_uuid: "7ef3ef9b6a7c449895a90ae689ebf8dc" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 42879 } } peers { permanent_uuid: "bc5b041ccd1545fa949bed034f8066b8" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40137 } }
I20260929 10:57:13.232578 25945 raft_consensus.cc:374] T eba7bac139fb4fafa1743db1f6661d09 P c8210aa708674cd58c8cc8a70d1dcf68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:57:13.232599 25945 raft_consensus.cc:729] T eba7bac139fb4fafa1743db1f6661d09 P c8210aa708674cd58c8cc8a70d1dcf68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8210aa708674cd58c8cc8a70d1dcf68, State: Initialized, Role: FOLLOWER
I20260929 10:57:13.232740 25945 consensus_queue.cc:260] T eba7bac139fb4fafa1743db1f6661d09 P c8210aa708674cd58c8cc8a70d1dcf68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8210aa708674cd58c8cc8a70d1dcf68" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38529 } } peers { permanent_uuid: "7ef3ef9b6a7c449895a90ae689ebf8dc" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 42879 } } peers { permanent_uuid: "bc5b041ccd1545fa949bed034f8066b8" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40137 } }
I20260929 10:57:13.233027 25945 ts_tablet_manager.cc:1428] T eba7bac139fb4fafa1743db1f6661d09 P c8210aa708674cd58c8cc8a70d1dcf68: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 10:57:13.233040 25937 heartbeater.cc:503] Master 127.10.238.254:40781 was elected leader, sending a full tablet report...
W20260929 10:57:13.304889 25802 tablet.cc:2369] T eba7bac139fb4fafa1743db1f6661d09 P bc5b041ccd1545fa949bed034f8066b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:57:13.325314 25947 raft_consensus.cc:482] T eba7bac139fb4fafa1743db1f6661d09 P c8210aa708674cd58c8cc8a70d1dcf68 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:57:13.325471 25947 raft_consensus.cc:504] T eba7bac139fb4fafa1743db1f6661d09 P c8210aa708674cd58c8cc8a70d1dcf68 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8210aa708674cd58c8cc8a70d1dcf68" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38529 } } peers { permanent_uuid: "7ef3ef9b6a7c449895a90ae689ebf8dc" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 42879 } } peers { permanent_uuid: "bc5b041ccd1545fa949bed034f8066b8" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40137 } }
I20260929 10:57:13.325905 25947 leader_election.cc:288] T eba7bac139fb4fafa1743db1f6661d09 P c8210aa708674cd58c8cc8a70d1dcf68 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ef3ef9b6a7c449895a90ae689ebf8dc (127.10.238.193:42879), bc5b041ccd1545fa949bed034f8066b8 (127.10.238.194:40137)
I20260929 10:57:13.329876 25670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:13.326148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:58589 (local address 127.10.238.193:42879)
0929 10:57:13.326197 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:13.326200 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:13.326349 (+ 149us) server_negotiation.cc:407] Connection header received
0929 10:57:13.326409 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:13.326410 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:13.326420 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:13.326475 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:57:13.326620 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:13.327385 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:13.328889 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:13.329487 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:13.329568 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:13.329578 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:13.329756 (+ 178us) server_negotiation.cc:299] Negotiation successful
0929 10:57:13.329807 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 10:57:13.329878 25807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:13.326173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:50951 (local address 127.10.238.194:40137)
0929 10:57:13.326220 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:13.326221 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:13.326230 (+ 9us) server_negotiation.cc:407] Connection header received
0929 10:57:13.326259 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:13.326260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:13.326291 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:13.326378 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:57:13.326609 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:13.327371 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:13.328888 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:13.329513 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:13.329575 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:13.329651 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:13.329778 (+ 127us) server_negotiation.cc:299] Negotiation successful
0929 10:57:13.329819 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 10:57:13.329873 25939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:13.326074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.194:40137 (local address 127.10.238.195:50951)
0929 10:57:13.326131 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:13.326143 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:13.326183 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:13.326404 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:57:13.326405 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:57:13.326406 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:57:13.326507 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:13.326509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:13.327465 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:13.327465 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:13.328789 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:13.328792 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:13.329581 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:13.329582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:13.329675 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:13.329676 (+ 1us) client_negotiation.cc:769] Sending connection context
0929 10:57:13.329734 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 10:57:13.329788 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":126}
I20260929 10:57:13.329885 25948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:13.326097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:42879 (local address 127.10.238.195:58589)
0929 10:57:13.326258 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:13.326269 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:13.326334 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:13.326492 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:57:13.326493 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:57:13.326494 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:57:13.326551 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:13.326553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:13.327464 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:13.327465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:13.328798 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:13.328799 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:13.329581 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:13.329582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:13.329672 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:13.329676 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 10:57:13.329734 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 10:57:13.329802 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":70,"thread_start_us":108,"threads_started":1}
I20260929 10:57:13.330257 25613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba7bac139fb4fafa1743db1f6661d09" candidate_uuid: "c8210aa708674cd58c8cc8a70d1dcf68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef3ef9b6a7c449895a90ae689ebf8dc" is_pre_election: true
I20260929 10:57:13.330394 25755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba7bac139fb4fafa1743db1f6661d09" candidate_uuid: "c8210aa708674cd58c8cc8a70d1dcf68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc5b041ccd1545fa949bed034f8066b8" is_pre_election: true
I20260929 10:57:13.330451 25613 raft_consensus.cc:2454] T eba7bac139fb4fafa1743db1f6661d09 P 7ef3ef9b6a7c449895a90ae689ebf8dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8210aa708674cd58c8cc8a70d1dcf68 in term 0.
I20260929 10:57:13.330543 25755 raft_consensus.cc:2454] T eba7bac139fb4fafa1743db1f6661d09 P bc5b041ccd1545fa949bed034f8066b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8210aa708674cd58c8cc8a70d1dcf68 in term 0.
I20260929 10:57:13.330713 25825 leader_election.cc:302] T eba7bac139fb4fafa1743db1f6661d09 P c8210aa708674cd58c8cc8a70d1dcf68 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ef3ef9b6a7c449895a90ae689ebf8dc, c8210aa708674cd58c8cc8a70d1dcf68; no voters:
I20260929 10:57:13.330864 25947 raft_consensus.cc:2781] T eba7bac139fb4fafa1743db1f6661d09 P c8210aa708674cd58c8cc8a70d1dcf68 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:57:13.330919 25947 raft_consensus.cc:482] T eba7bac139fb4fafa1743db1f6661d09 P c8210aa708674cd58c8cc8a70d1dcf68 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:57:13.330946 25947 raft_consensus.cc:3037] T eba7bac139fb4fafa1743db1f6661d09 P c8210aa708674cd58c8cc8a70d1dcf68 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:57:13.331759 25947 raft_consensus.cc:504] T eba7bac139fb4fafa1743db1f6661d09 P c8210aa708674cd58c8cc8a70d1dcf68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8210aa708674cd58c8cc8a70d1dcf68" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38529 } } peers { permanent_uuid: "7ef3ef9b6a7c449895a90ae689ebf8dc" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 42879 } } peers { permanent_uuid: "bc5b041ccd1545fa949bed034f8066b8" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40137 } }
I20260929 10:57:13.331905 25947 leader_election.cc:288] T eba7bac139fb4fafa1743db1f6661d09 P c8210aa708674cd58c8cc8a70d1dcf68 [CANDIDATE]: Term 1 election: Requested vote from peers 7ef3ef9b6a7c449895a90ae689ebf8dc (127.10.238.193:42879), bc5b041ccd1545fa949bed034f8066b8 (127.10.238.194:40137)
I20260929 10:57:13.332056 25613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba7bac139fb4fafa1743db1f6661d09" candidate_uuid: "c8210aa708674cd58c8cc8a70d1dcf68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef3ef9b6a7c449895a90ae689ebf8dc"
I20260929 10:57:13.332057 25755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba7bac139fb4fafa1743db1f6661d09" candidate_uuid: "c8210aa708674cd58c8cc8a70d1dcf68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc5b041ccd1545fa949bed034f8066b8"
I20260929 10:57:13.332135 25755 raft_consensus.cc:3037] T eba7bac139fb4fafa1743db1f6661d09 P bc5b041ccd1545fa949bed034f8066b8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:57:13.332136 25613 raft_consensus.cc:3037] T eba7bac139fb4fafa1743db1f6661d09 P 7ef3ef9b6a7c449895a90ae689ebf8dc [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:57:13.333122 25755 raft_consensus.cc:2454] T eba7bac139fb4fafa1743db1f6661d09 P bc5b041ccd1545fa949bed034f8066b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8210aa708674cd58c8cc8a70d1dcf68 in term 1.
I20260929 10:57:13.333359 25826 leader_election.cc:302] T eba7bac139fb4fafa1743db1f6661d09 P c8210aa708674cd58c8cc8a70d1dcf68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc5b041ccd1545fa949bed034f8066b8, c8210aa708674cd58c8cc8a70d1dcf68; no voters:
I20260929 10:57:13.333488 25613 raft_consensus.cc:2454] T eba7bac139fb4fafa1743db1f6661d09 P 7ef3ef9b6a7c449895a90ae689ebf8dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8210aa708674cd58c8cc8a70d1dcf68 in term 1.
I20260929 10:57:13.333491 25947 raft_consensus.cc:2781] T eba7bac139fb4fafa1743db1f6661d09 P c8210aa708674cd58c8cc8a70d1dcf68 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:57:13.333675 25947 raft_consensus.cc:686] T eba7bac139fb4fafa1743db1f6661d09 P c8210aa708674cd58c8cc8a70d1dcf68 [term 1 LEADER]: Becoming Leader. State: Replica: c8210aa708674cd58c8cc8a70d1dcf68, State: Running, Role: LEADER
I20260929 10:57:13.333787 25947 consensus_queue.cc:237] T eba7bac139fb4fafa1743db1f6661d09 P c8210aa708674cd58c8cc8a70d1dcf68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8210aa708674cd58c8cc8a70d1dcf68" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38529 } } peers { permanent_uuid: "7ef3ef9b6a7c449895a90ae689ebf8dc" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 42879 } } peers { permanent_uuid: "bc5b041ccd1545fa949bed034f8066b8" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40137 } }
I20260929 10:57:13.334662 25473 catalog_manager.cc:5495] T eba7bac139fb4fafa1743db1f6661d09 P c8210aa708674cd58c8cc8a70d1dcf68 reported cstate change: term changed from 0 to 1, leader changed from <none> to c8210aa708674cd58c8cc8a70d1dcf68 (127.10.238.195). New cstate: current_term: 1 leader_uuid: "c8210aa708674cd58c8cc8a70d1dcf68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8210aa708674cd58c8cc8a70d1dcf68" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ef3ef9b6a7c449895a90ae689ebf8dc" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc5b041ccd1545fa949bed034f8066b8" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40137 } health_report { overall_health: UNKNOWN } } }
I20260929 10:57:13.339339 25942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:13.337361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45062 (local address 127.10.238.195:38529)
0929 10:57:13.337401 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:13.337403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:13.337414 (+ 11us) server_negotiation.cc:407] Connection header received
0929 10:57:13.337490 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:13.337491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:13.337505 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:13.337549 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:57:13.337630 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:13.338070 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:13.338745 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:13.338920 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:13.338969 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:13.339098 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:57:13.339182 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:57:13.339245 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:13.339274 (+ 29us) server_negotiation.cc:299] Negotiation successful
0929 10:57:13.339291 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 10:57:13.340999 25613 raft_consensus.cc:1261] T eba7bac139fb4fafa1743db1f6661d09 P 7ef3ef9b6a7c449895a90ae689ebf8dc [term 1 FOLLOWER]: Refusing update from remote peer c8210aa708674cd58c8cc8a70d1dcf68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:57:13.340999 25755 raft_consensus.cc:1261] T eba7bac139fb4fafa1743db1f6661d09 P bc5b041ccd1545fa949bed034f8066b8 [term 1 FOLLOWER]: Refusing update from remote peer c8210aa708674cd58c8cc8a70d1dcf68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:57:13.341393 25949 consensus_queue.cc:1035] T eba7bac139fb4fafa1743db1f6661d09 P c8210aa708674cd58c8cc8a70d1dcf68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc5b041ccd1545fa949bed034f8066b8" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:57:13.341534 25947 consensus_queue.cc:1035] T eba7bac139fb4fafa1743db1f6661d09 P c8210aa708674cd58c8cc8a70d1dcf68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef3ef9b6a7c449895a90ae689ebf8dc" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:57:13.343531 25951 mvcc.cc:205] Tried to move back new op lower bound from 7334622958962221056 to 7334622958936023040. Current Snapshot: MvccSnapshot[applied={T|T < 7334622958962221056}]
I20260929 10:57:13.343767 25952 mvcc.cc:205] Tried to move back new op lower bound from 7334622958962221056 to 7334622958936023040. Current Snapshot: MvccSnapshot[applied={T|T < 7334622958962221056}]
I20260929 10:57:13.344437 25953 mvcc.cc:205] Tried to move back new op lower bound from 7334622958962221056 to 7334622958936023040. Current Snapshot: MvccSnapshot[applied={T|T < 7334622958962221056 or (T in {7334622958962221056})}]
I20260929 10:57:13.345721 25473 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:45832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:57:13.345822 25473 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:45832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:57:13.346551 25473 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321: Sending DeleteTablet for 3 replicas of tablet eba7bac139fb4fafa1743db1f6661d09
I20260929 10:57:13.346918 25735 tablet_service.cc:1514] Processing DeleteTablet for tablet eba7bac139fb4fafa1743db1f6661d09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:57:13 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:56502
I20260929 10:57:13.346940 25871 tablet_service.cc:1514] Processing DeleteTablet for tablet eba7bac139fb4fafa1743db1f6661d09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:57:13 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:45056
I20260929 10:57:13.347010 25808 tablet_replica.cc:330] T eba7bac139fb4fafa1743db1f6661d09 P bc5b041ccd1545fa949bed034f8066b8: stopping tablet replica
I20260929 10:57:13.347028 25578 tablet_service.cc:1514] Processing DeleteTablet for tablet eba7bac139fb4fafa1743db1f6661d09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:57:13 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:55864
I20260929 10:57:13.347059 25808 raft_consensus.cc:2229] T eba7bac139fb4fafa1743db1f6661d09 P bc5b041ccd1545fa949bed034f8066b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:57:13.347097 25808 raft_consensus.cc:2258] T eba7bac139fb4fafa1743db1f6661d09 P bc5b041ccd1545fa949bed034f8066b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:57:13.347110 25671 tablet_replica.cc:330] T eba7bac139fb4fafa1743db1f6661d09 P 7ef3ef9b6a7c449895a90ae689ebf8dc: stopping tablet replica
I20260929 10:57:13.347160 25671 raft_consensus.cc:2229] T eba7bac139fb4fafa1743db1f6661d09 P 7ef3ef9b6a7c449895a90ae689ebf8dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:57:13.347194 25671 raft_consensus.cc:2258] T eba7bac139fb4fafa1743db1f6661d09 P 7ef3ef9b6a7c449895a90ae689ebf8dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:57:13.347290 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 25512
I20260929 10:57:13.347381 25960 tablet_replica.cc:330] T eba7bac139fb4fafa1743db1f6661d09 P c8210aa708674cd58c8cc8a70d1dcf68: stopping tablet replica
I20260929 10:57:13.347474 25960 raft_consensus.cc:2229] T eba7bac139fb4fafa1743db1f6661d09 P c8210aa708674cd58c8cc8a70d1dcf68 [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:57:13.347539 25671 ts_tablet_manager.cc:1905] T eba7bac139fb4fafa1743db1f6661d09 P 7ef3ef9b6a7c449895a90ae689ebf8dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:57:13.347558 25960 raft_consensus.cc:2258] T eba7bac139fb4fafa1743db1f6661d09 P c8210aa708674cd58c8cc8a70d1dcf68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:57:13.348395 25960 ts_tablet_manager.cc:1905] T eba7bac139fb4fafa1743db1f6661d09 P c8210aa708674cd58c8cc8a70d1dcf68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:57:13.349785 25960 ts_tablet_manager.cc:1918] T eba7bac139fb4fafa1743db1f6661d09 P c8210aa708674cd58c8cc8a70d1dcf68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:57:13.349875 25960 log.cc:1196] T eba7bac139fb4fafa1743db1f6661d09 P c8210aa708674cd58c8cc8a70d1dcf68: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-2/wal/wals/eba7bac139fb4fafa1743db1f6661d09
I20260929 10:57:13.350248 25960 ts_tablet_manager.cc:1939] T eba7bac139fb4fafa1743db1f6661d09 P c8210aa708674cd58c8cc8a70d1dcf68: Deleting consensus metadata
I20260929 10:57:13.350270 25808 ts_tablet_manager.cc:1905] T eba7bac139fb4fafa1743db1f6661d09 P bc5b041ccd1545fa949bed034f8066b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:57:13.350726 25458 catalog_manager.cc:4841] TS c8210aa708674cd58c8cc8a70d1dcf68 (127.10.238.195:38529): tablet eba7bac139fb4fafa1743db1f6661d09 (table test-table [id=567c3a823ced44b18d73aa912e9ba6a9]) successfully deleted
I20260929 10:57:13.351593 25808 ts_tablet_manager.cc:1918] T eba7bac139fb4fafa1743db1f6661d09 P bc5b041ccd1545fa949bed034f8066b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:57:13.351655 25808 log.cc:1196] T eba7bac139fb4fafa1743db1f6661d09 P bc5b041ccd1545fa949bed034f8066b8: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-1/wal/wals/eba7bac139fb4fafa1743db1f6661d09
I20260929 10:57:13.351867 25808 ts_tablet_manager.cc:1939] T eba7bac139fb4fafa1743db1f6661d09 P bc5b041ccd1545fa949bed034f8066b8: Deleting consensus metadata
I20260929 10:57:13.352206 25460 catalog_manager.cc:4841] TS bc5b041ccd1545fa949bed034f8066b8 (127.10.238.194:40137): tablet eba7bac139fb4fafa1743db1f6661d09 (table test-table [id=567c3a823ced44b18d73aa912e9ba6a9]) successfully deleted
W20260929 10:57:13.354457 25459 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.238.193:42879 (error 108)
I20260929 10:57:13.354839 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 25667
I20260929 10:57:13.354970 25669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:13.354834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:42879 (local address 127.0.0.1:55878)
0929 10:57:13.354891 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:13.354935 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.193:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 10:57:13.355492 25459 catalog_manager.cc:4584] TS 7ef3ef9b6a7c449895a90ae689ebf8dc (127.10.238.193:42879): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eba7bac139fb4fafa1743db1f6661d09: Network error: Client connection negotiation failed: client connection to 127.10.238.193:42879: connect: Connection refused (error 111)
I20260929 10:57:13.362355 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 25805
I20260929 10:57:13.367707 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 25441
I20260929 10:57:13.372869 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:40781
--webserver_interface=127.10.238.254
--webserver_port=40077
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:42761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:40781
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:57:13.455148 25962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:57:13.455358 25962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:57:13.455380 25962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:57:13.456576 25962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:57:13.456619 25962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:57:13.456633 25962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:57:13.456646 25962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:57:13.456698 25962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:57:13.458277 25962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:42761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:40781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:40781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.238.254
--webserver_port=40077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:57:13.458541 25962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:57:13.458776 25962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:57:13.463582 25967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:13.463620 25970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:13.463626 25968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:57:13.463737 25962 server_base.cc:1034] running on GCE node
I20260929 10:57:13.464082 25962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:57:13.464371 25962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:57:13.465531 25962 hybrid_clock.cc:648] HybridClock initialized: now 1790679433465497 us; error 46 us; skew 500 ppm
Sep 29 10:57:13 dist-test-slave-8nrt krb5kdc[24896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679433, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:57:13.467923 25962 init.cc:377] Logged in from keytab as kudu/127.10.238.254@KRBTEST.COM (short username kudu)
I20260929 10:57:13.468392 25962 webserver.cc:460] Webserver started at http://127.10.238.254:40077/ using document root <none> and password file <none>
I20260929 10:57:13.468619 25962 fs_manager.cc:359] Metadata directory not provided
I20260929 10:57:13.468688 25962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:57:13.470057 25962 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:57:13.470698 25977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:13.470876 25962 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:57:13.470975 25962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "98271e7b5d3848a7bd8941df8d73c321"
format_stamp: "Formatted at 2026-09-29 10:57:11 on dist-test-slave-8nrt"
server_key: "9f0450d2f78c5552cf3ed8c7e823e9de"
server_key_iv: "d0ca58d493c8b26db9b3e66341dcdc06"
server_key_version: "encryptionkey@0"
I20260929 10:57:13.471320 25962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:57:13.481559 25962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:57:13.483736 25962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:57:13.483899 25962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:57:13.487895 25962 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.254:40781
I20260929 10:57:13.487942 26029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.254:40781 every 8 connection(s)
I20260929 10:57:13.488335 25962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:57:13.489920 26030 sys_catalog.cc:263] Verifying existing consensus state
I20260929 10:57:13.490568 26030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321: Bootstrap starting.
I20260929 10:57:13.497680 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 25962
I20260929 10:57:13.497990 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.193:42879
--local_ip_for_outbound_sockets=127.10.238.193
--tserver_master_addrs=127.10.238.254:40781
--webserver_port=40477
--webserver_interface=127.10.238.193
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab
--principal=kudu/127.10.238.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:42761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:57:13.499415 26030 log.cc:824] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321: Log is configured to *not* fsync() on all Append() calls
I20260929 10:57:13.501812 26030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 10:57:13.502106 26030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321: Bootstrap complete.
I20260929 10:57:13.504347 26030 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98271e7b5d3848a7bd8941df8d73c321" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 40781 } }
I20260929 10:57:13.504595 26030 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98271e7b5d3848a7bd8941df8d73c321, State: Initialized, Role: FOLLOWER
I20260929 10:57:13.504793 26030 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98271e7b5d3848a7bd8941df8d73c321" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 40781 } }
I20260929 10:57:13.504891 26030 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:57:13.504946 26030 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:57:13.504995 26030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321 [term 2 FOLLOWER]: Advancing to term 3
I20260929 10:57:13.505999 26030 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98271e7b5d3848a7bd8941df8d73c321" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 40781 } }
I20260929 10:57:13.506119 26030 leader_election.cc:302] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98271e7b5d3848a7bd8941df8d73c321; no voters:
I20260929 10:57:13.506331 26030 leader_election.cc:288] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 10:57:13.506371 26034 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 10:57:13.506474 26034 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321 [term 3 LEADER]: Becoming Leader. State: Replica: 98271e7b5d3848a7bd8941df8d73c321, State: Running, Role: LEADER
I20260929 10:57:13.506593 26030 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:57:13.506624 26034 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98271e7b5d3848a7bd8941df8d73c321" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 40781 } }
I20260929 10:57:13.507026 26035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "98271e7b5d3848a7bd8941df8d73c321" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98271e7b5d3848a7bd8941df8d73c321" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 40781 } } }
I20260929 10:57:13.507117 26035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321 [sys.catalog]: This master's current role is: LEADER
I20260929 10:57:13.507381 26043 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:57:13.507369 26036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98271e7b5d3848a7bd8941df8d73c321. Latest consensus state: current_term: 3 leader_uuid: "98271e7b5d3848a7bd8941df8d73c321" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98271e7b5d3848a7bd8941df8d73c321" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 40781 } } }
I20260929 10:57:13.508179 26036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98271e7b5d3848a7bd8941df8d73c321 [sys.catalog]: This master's current role is: LEADER
I20260929 10:57:13.508369 26043 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:57:13.508469 26043 catalog_manager.cc:1259] Loaded cluster ID: 5af86ed08eff4ad587bf0a4e20da6be3
I20260929 10:57:13.508493 26043 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260929 10:57:13.508836 26043 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f20b4dca9ad google::LogMessage::Fail() at ??:0
@ 0x7f20b4dcd88a google::LogMessage::SendToLog() at ??:0
@ 0x7f20b4dca4de google::LogMessage::Flush() at ??:0
@ 0x7f20b4dcba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f20bcceb0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f20bccf48b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f20b6c980d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f20b6c8d954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f20b7af16db start_thread at ??:0
@ 0x7f20b38c861f clone at ??:0
W20260929 10:57:13.609638 26032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:57:13.609851 26032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:57:13.609877 26032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:57:13.611027 26032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:57:13.611073 26032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:57:13.611101 26032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.193
I20260929 10:57:13.612566 26032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:42761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab
--principal=kudu/127.10.238.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.193:42879
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.238.193
--webserver_port=40477
--tserver_master_addrs=127.10.238.254:40781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.193
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:57:13.612810 26032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:57:13.613039 26032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:57:13.618280 26069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:57:13.618392 26032 server_base.cc:1034] running on GCE node
W20260929 10:57:13.618314 26067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:13.618505 26066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:57:13.618764 26032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:57:13.619020 26032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:57:13.620226 26032 hybrid_clock.cc:648] HybridClock initialized: now 1790679433620199 us; error 41 us; skew 500 ppm
Sep 29 10:57:13 dist-test-slave-8nrt krb5kdc[24896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679433, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:57:13.622947 26032 init.cc:377] Logged in from keytab as kudu/127.10.238.193@KRBTEST.COM (short username kudu)
I20260929 10:57:13.623387 26032 webserver.cc:460] Webserver started at http://127.10.238.193:40477/ using document root <none> and password file <none>
I20260929 10:57:13.623602 26032 fs_manager.cc:359] Metadata directory not provided
I20260929 10:57:13.623648 26032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:57:13.625046 26032 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:57:13.626060 26076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:13.626268 26032 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 10:57:13.626366 26032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-0/wal
uuid: "7ef3ef9b6a7c449895a90ae689ebf8dc"
format_stamp: "Formatted at 2026-09-29 10:57:11 on dist-test-slave-8nrt"
server_key: "0f1b2a95b8a5ff390552d34e3929493d"
server_key_iv: "8f4287606c955b0856f2b0a758722d5f"
server_key_version: "encryptionkey@0"
I20260929 10:57:13.626703 26032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:57:13.652395 26032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:57:13.655072 26032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:57:13.655256 26032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:57:13.655536 26032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:57:13.656044 26083 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 10:57:13.658011 26032 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 10:57:13.658095 26032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260929 10:57:13.658164 26032 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 10:57:13.658836 26032 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 10:57:13.658895 26032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260929 10:57:13.658977 26083 tablet_bootstrap.cc:492] T eba7bac139fb4fafa1743db1f6661d09 P 7ef3ef9b6a7c449895a90ae689ebf8dc: Bootstrap starting.
I20260929 10:57:13.660745 26083 log.cc:824] T eba7bac139fb4fafa1743db1f6661d09 P 7ef3ef9b6a7c449895a90ae689ebf8dc: Log is configured to *not* fsync() on all Append() calls
I20260929 10:57:13.663800 26083 tablet_bootstrap.cc:492] T eba7bac139fb4fafa1743db1f6661d09 P 7ef3ef9b6a7c449895a90ae689ebf8dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 10:57:13.664098 26083 tablet_bootstrap.cc:492] T eba7bac139fb4fafa1743db1f6661d09 P 7ef3ef9b6a7c449895a90ae689ebf8dc: Bootstrap complete.
I20260929 10:57:13.664240 26083 ts_tablet_manager.cc:1397] T eba7bac139fb4fafa1743db1f6661d09 P 7ef3ef9b6a7c449895a90ae689ebf8dc: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260929 10:57:13.666288 26032 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.193:42879
I20260929 10:57:13.666321 26191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.193:42879 every 8 connection(s)
I20260929 10:57:13.666225 26083 raft_consensus.cc:348] T eba7bac139fb4fafa1743db1f6661d09 P 7ef3ef9b6a7c449895a90ae689ebf8dc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8210aa708674cd58c8cc8a70d1dcf68" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38529 } } peers { permanent_uuid: "7ef3ef9b6a7c449895a90ae689ebf8dc" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 42879 } } peers { permanent_uuid: "bc5b041ccd1545fa949bed034f8066b8" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40137 } }
I20260929 10:57:13.666581 26083 raft_consensus.cc:729] T eba7bac139fb4fafa1743db1f6661d09 P 7ef3ef9b6a7c449895a90ae689ebf8dc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ef3ef9b6a7c449895a90ae689ebf8dc, State: Initialized, Role: FOLLOWER
I20260929 10:57:13.666617 26032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
I20260929 10:57:13.666710 26083 consensus_queue.cc:260] T eba7bac139fb4fafa1743db1f6661d09 P 7ef3ef9b6a7c449895a90ae689ebf8dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8210aa708674cd58c8cc8a70d1dcf68" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38529 } } peers { permanent_uuid: "7ef3ef9b6a7c449895a90ae689ebf8dc" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 42879 } } peers { permanent_uuid: "bc5b041ccd1545fa949bed034f8066b8" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40137 } }
I20260929 10:57:13.667215 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 26032
I20260929 10:57:13.667537 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.194:40137
--local_ip_for_outbound_sockets=127.10.238.194
--tserver_master_addrs=127.10.238.254:40781
--webserver_port=44967
--webserver_interface=127.10.238.194
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab
--principal=kudu/127.10.238.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:42761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:57:13.666966 26083 ts_tablet_manager.cc:1428] T eba7bac139fb4fafa1743db1f6661d09 P 7ef3ef9b6a7c449895a90ae689ebf8dc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260929 10:57:13.752884 26196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:57:13.753108 26196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:57:13.753149 26196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:57:13.754351 26196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:57:13.754418 26196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:57:13.754464 26196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.194
I20260929 10:57:13.755944 26196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:42761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab
--principal=kudu/127.10.238.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.194:40137
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.238.194
--webserver_port=44967
--tserver_master_addrs=127.10.238.254:40781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.194
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:57:13.756204 26196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:57:13.756445 26196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:57:13.761457 26204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:13.761461 26202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:13.761551 26201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:57:13.761716 26196 server_base.cc:1034] running on GCE node
I20260929 10:57:13.762077 26196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:57:13.762396 26196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:57:13.763557 26196 hybrid_clock.cc:648] HybridClock initialized: now 1790679433763545 us; error 28 us; skew 500 ppm
Sep 29 10:57:13 dist-test-slave-8nrt krb5kdc[24896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679433, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:57:13.766155 26196 init.cc:377] Logged in from keytab as kudu/127.10.238.194@KRBTEST.COM (short username kudu)
I20260929 10:57:13.766597 26196 webserver.cc:460] Webserver started at http://127.10.238.194:44967/ using document root <none> and password file <none>
I20260929 10:57:13.766829 26196 fs_manager.cc:359] Metadata directory not provided
I20260929 10:57:13.766875 26196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:57:13.768273 26196 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 10:57:13.768954 26211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:13.769146 26196 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 10:57:13.769232 26196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-1/wal
uuid: "bc5b041ccd1545fa949bed034f8066b8"
format_stamp: "Formatted at 2026-09-29 10:57:12 on dist-test-slave-8nrt"
server_key: "13fec2abed2246ad8a5095a75cf58d03"
server_key_iv: "71800f02a4a8bebedbbad49061ba5d8c"
server_key_version: "encryptionkey@0"
I20260929 10:57:13.769615 26196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:57:13.781829 26196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:57:13.783936 26196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:57:13.784080 26196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:57:13.784314 26196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:57:13.784626 26196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:57:13.784678 26196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:13.784718 26196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:57:13.784746 26196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:13.791002 26196 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.194:40137
I20260929 10:57:13.791050 26324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.194:40137 every 8 connection(s)
I20260929 10:57:13.791402 26196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
I20260929 10:57:13.792721 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 26196
I20260929 10:57:13.793062 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.195:38529
--local_ip_for_outbound_sockets=127.10.238.195
--tserver_master_addrs=127.10.238.254:40781
--webserver_port=41805
--webserver_interface=127.10.238.195
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab
--principal=kudu/127.10.238.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:42761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:57:13.879726 26195 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:57:13.667342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.193:57227)
0929 10:57:13.667529 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:13.667577 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:13.667750 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:13.879493 (+211743us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:40781: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":89,"threads_started":1}
W20260929 10:57:13.880043 26192 heartbeater.cc:644] Failed to heartbeat to 127.10.238.254:40781 (0 consecutive failures): Network error: Failed to ping master at 127.10.238.254:40781: Client connection negotiation failed: client connection to 127.10.238.254:40781: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260929 10:57:13.880137 26327 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:57:13.792071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.194:38755)
0929 10:57:13.792215 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:13.792260 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:13.792882 (+ 622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:13.879479 (+ 86597us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:40781: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":67,"threads_started":1}
I20260929 10:57:13.880443 26195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:13.880297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.193:43859)
0929 10:57:13.880369 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:13.880414 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:40781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260929 10:57:13.880581 26325 heartbeater.cc:644] Failed to heartbeat to 127.10.238.254:40781 (0 consecutive failures): Network error: Failed to ping master at 127.10.238.254:40781: Client connection negotiation failed: client connection to 127.10.238.254:40781: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 10:57:13.880823 26195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:13.880695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.193:50371)
0929 10:57:13.880760 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:13.880796 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:40781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 10:57:13.880864 26327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:13.880760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.194:59605)
0929 10:57:13.880804 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:13.880846 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:40781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260929 10:57:13.880965 26192 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 10:57:13.881215 26327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:13.881115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.194:58459)
0929 10:57:13.881168 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:13.881199 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:40781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260929 10:57:13.881358 26325 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 10:57:13.896605 26328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:57:13.896875 26328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:57:13.896921 26328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:57:13.898478 26328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:57:13.898562 26328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:57:13.898617 26328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.195
I20260929 10:57:13.900309 26328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:42761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab
--principal=kudu/127.10.238.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.195:38529
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.238.195
--webserver_port=41805
--tserver_master_addrs=127.10.238.254:40781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.195
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:57:13.900606 26328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:57:13.900885 26328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:57:13.906518 26334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:13.906495 26333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:13.906522 26336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:57:13.906664 26328 server_base.cc:1034] running on GCE node
I20260929 10:57:13.906966 26328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:57:13.907164 26328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:57:13.908299 26328 hybrid_clock.cc:648] HybridClock initialized: now 1790679433908292 us; error 29 us; skew 500 ppm
Sep 29 10:57:13 dist-test-slave-8nrt krb5kdc[24896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679433, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:57:13.910823 26328 init.cc:377] Logged in from keytab as kudu/127.10.238.195@KRBTEST.COM (short username kudu)
I20260929 10:57:13.911317 26328 webserver.cc:460] Webserver started at http://127.10.238.195:41805/ using document root <none> and password file <none>
I20260929 10:57:13.911566 26328 fs_manager.cc:359] Metadata directory not provided
I20260929 10:57:13.911610 26328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:57:13.912969 26328 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:57:13.913615 26343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:13.913793 26328 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:13.913884 26328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-2/wal
uuid: "c8210aa708674cd58c8cc8a70d1dcf68"
format_stamp: "Formatted at 2026-09-29 10:57:12 on dist-test-slave-8nrt"
server_key: "4953c7c4c36d2636e09b055834e005fb"
server_key_iv: "1dbddcd4b64f72137179b0316ce287b8"
server_key_version: "encryptionkey@0"
I20260929 10:57:13.914237 26328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:57:13.939661 26328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:57:13.941725 26328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:57:13.941876 26328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:57:13.942144 26328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:57:13.942464 26328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:57:13.942498 26328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:13.942520 26328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:57:13.942567 26328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:13.948467 26328 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.195:38529
I20260929 10:57:13.948518 26456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.195:38529 every 8 connection(s)
I20260929 10:57:13.948870 26328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
I20260929 10:57:13.949980 26459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:13.949601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.195:41933)
0929 10:57:13.949759 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:13.949840 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:40781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":75,"threads_started":1}
W20260929 10:57:13.950376 26457 heartbeater.cc:644] Failed to heartbeat to 127.10.238.254:40781 (0 consecutive failures): Network error: Failed to ping master at 127.10.238.254:40781: Client connection negotiation failed: client connection to 127.10.238.254:40781: connect: Connection refused (error 111)
I20260929 10:57:13.950659 26459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:13.950570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.195:35187)
0929 10:57:13.950615 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:13.950641 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:40781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 10:57:13.950897 26459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:13.950846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.195:41713)
0929 10:57:13.950867 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:13.950885 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:40781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260929 10:57:13.951002 26457 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 10:57:13.958850 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 26328
I20260929 10:57:14.881763 26461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:14.881419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.193:51973)
0929 10:57:14.881646 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:14.881692 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:40781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":110,"threads_started":1}
I20260929 10:57:14.882020 26462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:14.881745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.194:41289)
0929 10:57:14.881909 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:14.881961 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:40781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":64,"threads_started":1}
I20260929 10:57:14.951578 26463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:14.951358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.195:34769)
0929 10:57:14.951492 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:14.951529 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:40781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":71,"threads_started":1}
I20260929 10:57:15.462234 26464 raft_consensus.cc:482] T eba7bac139fb4fafa1743db1f6661d09 P 7ef3ef9b6a7c449895a90ae689ebf8dc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:57:15.462445 26464 raft_consensus.cc:504] T eba7bac139fb4fafa1743db1f6661d09 P 7ef3ef9b6a7c449895a90ae689ebf8dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8210aa708674cd58c8cc8a70d1dcf68" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38529 } } peers { permanent_uuid: "7ef3ef9b6a7c449895a90ae689ebf8dc" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 42879 } } peers { permanent_uuid: "bc5b041ccd1545fa949bed034f8066b8" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40137 } }
I20260929 10:57:15.462913 26464 leader_election.cc:288] T eba7bac139fb4fafa1743db1f6661d09 P 7ef3ef9b6a7c449895a90ae689ebf8dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c8210aa708674cd58c8cc8a70d1dcf68 (127.10.238.195:38529), bc5b041ccd1545fa949bed034f8066b8 (127.10.238.194:40137)
Sep 29 10:57:15 dist-test-slave-8nrt krb5kdc[24896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679433, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.193@KRBTEST.COM for kudu/127.10.238.195@KRBTEST.COM
Sep 29 10:57:15 dist-test-slave-8nrt krb5kdc[24896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679433, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.193@KRBTEST.COM for kudu/127.10.238.194@KRBTEST.COM
I20260929 10:57:15.473135 26465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:15.463029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.195:38529 (local address 127.10.238.193:35511)
0929 10:57:15.463187 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:15.463198 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:15.463256 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:15.464257 (+ 1001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:57:15.464258 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:57:15.464723 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:57:15.464814 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:15.464816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:15.465959 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:15.465960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:15.466504 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:15.466506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:15.467113 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:15.467114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:15.467181 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:15.467794 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:57:15.467799 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:57:15.469312 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:57:15.471442 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:15.471450 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:15.471452 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:15.471804 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:15.472108 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:15.472109 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:15.472109 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:15.472192 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:15.472660 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:57:15.472667 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:57:15.472830 (+ 163us) client_negotiation.cc:769] Sending connection context
0929 10:57:15.472938 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 10:57:15.473054 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":71,"threads_started":1}
I20260929 10:57:15.473207 26467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:15.463191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:35511 (local address 127.10.238.195:38529)
0929 10:57:15.463406 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:15.463416 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:15.463434 (+ 18us) server_negotiation.cc:407] Connection header received
0929 10:57:15.464053 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:15.464057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:15.464109 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:15.464269 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:57:15.464940 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:15.465859 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:15.466603 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:15.467029 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:15.467145 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:15.469417 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:57:15.469421 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:57:15.469421 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:57:15.469431 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:57:15.471355 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:15.471913 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:15.471916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:15.471916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:15.472012 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:15.472269 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:15.472270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:15.472270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:15.472580 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:57:15.472665 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:15.472997 (+ 332us) server_negotiation.cc:299] Negotiation successful
0929 10:57:15.473097 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":74,"threads_started":1}
I20260929 10:57:15.473591 26466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:15.463028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.194:40137 (local address 127.10.238.193:36615)
0929 10:57:15.463229 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:15.463242 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:15.463313 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:15.463719 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:57:15.463721 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:57:15.464092 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:57:15.464392 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:15.464399 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:15.465638 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:15.465639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:15.466376 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:15.466378 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:15.467021 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:15.467022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:15.467149 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:15.467697 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:57:15.467706 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:57:15.469695 (+ 1989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:57:15.471505 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:15.471506 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:15.471506 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:15.471806 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:15.472067 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:15.472068 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:15.472068 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:15.472196 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:15.473241 (+ 1045us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:57:15.473242 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:57:15.473303 (+ 61us) client_negotiation.cc:769] Sending connection context
0929 10:57:15.473377 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 10:57:15.473468 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":21,"thread_start_us":42,"threads_started":1}
I20260929 10:57:15.474012 26468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:15.463247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:36615 (local address 127.10.238.194:40137)
0929 10:57:15.463373 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:15.463378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:15.463390 (+ 12us) server_negotiation.cc:407] Connection header received
0929 10:57:15.463493 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:15.463494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:15.463524 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:15.463596 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:57:15.464526 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:15.465527 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:15.466478 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:15.466916 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:15.467047 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:15.469813 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:57:15.469817 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:57:15.469817 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:57:15.469827 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:57:15.471421 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:15.471893 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:15.471897 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:15.471897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:15.471998 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:15.472268 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:15.472269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:15.472269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:15.472588 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:57:15.472684 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:15.473741 (+ 1057us) server_negotiation.cc:299] Negotiation successful
0929 10:57:15.473886 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":67,"threads_started":1}
W20260929 10:57:15.474076 26077 leader_election.cc:341] T eba7bac139fb4fafa1743db1f6661d09 P 7ef3ef9b6a7c449895a90ae689ebf8dc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c8210aa708674cd58c8cc8a70d1dcf68 (127.10.238.195:38529): Not found: Tablet not found: eba7bac139fb4fafa1743db1f6661d09
W20260929 10:57:15.474611 26079 leader_election.cc:341] T eba7bac139fb4fafa1743db1f6661d09 P 7ef3ef9b6a7c449895a90ae689ebf8dc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bc5b041ccd1545fa949bed034f8066b8 (127.10.238.194:40137): Not found: Tablet not found: eba7bac139fb4fafa1743db1f6661d09
I20260929 10:57:15.474699 26079 leader_election.cc:302] T eba7bac139fb4fafa1743db1f6661d09 P 7ef3ef9b6a7c449895a90ae689ebf8dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ef3ef9b6a7c449895a90ae689ebf8dc; no voters: bc5b041ccd1545fa949bed034f8066b8, c8210aa708674cd58c8cc8a70d1dcf68
I20260929 10:57:15.474799 26464 raft_consensus.cc:2726] T eba7bac139fb4fafa1743db1f6661d09 P 7ef3ef9b6a7c449895a90ae689ebf8dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 10:57:15.882395 26466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:15.882210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.193:37651)
0929 10:57:15.882269 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:15.882339 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:40781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260929 10:57:15.882781 26469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:15.882518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.194:45299)
0929 10:57:15.882670 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:15.882722 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:40781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":73,"threads_started":1}
I20260929 10:57:15.952277 26470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:15.952059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.195:40349)
0929 10:57:15.952187 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:15.952234 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:40781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":56,"threads_started":1}
I20260929 10:57:16.883157 26471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:16.882914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.193:43191)
0929 10:57:16.883052 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:16.883086 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:40781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":64,"threads_started":1}
I20260929 10:57:16.883467 26472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:16.883253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.194:45215)
0929 10:57:16.883381 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:16.883427 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:40781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":84,"threads_started":1}
I20260929 10:57:16.952992 26473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:16.952771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.195:51239)
0929 10:57:16.952913 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:16.952952 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:40781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":85,"threads_started":1}
I20260929 10:57:17.464980 26474 raft_consensus.cc:482] T eba7bac139fb4fafa1743db1f6661d09 P 7ef3ef9b6a7c449895a90ae689ebf8dc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:57:17.465077 26474 raft_consensus.cc:504] T eba7bac139fb4fafa1743db1f6661d09 P 7ef3ef9b6a7c449895a90ae689ebf8dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8210aa708674cd58c8cc8a70d1dcf68" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38529 } } peers { permanent_uuid: "7ef3ef9b6a7c449895a90ae689ebf8dc" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 42879 } } peers { permanent_uuid: "bc5b041ccd1545fa949bed034f8066b8" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40137 } }
I20260929 10:57:17.465301 26474 leader_election.cc:288] T eba7bac139fb4fafa1743db1f6661d09 P 7ef3ef9b6a7c449895a90ae689ebf8dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c8210aa708674cd58c8cc8a70d1dcf68 (127.10.238.195:38529), bc5b041ccd1545fa949bed034f8066b8 (127.10.238.194:40137)
W20260929 10:57:17.465749 26077 leader_election.cc:341] T eba7bac139fb4fafa1743db1f6661d09 P 7ef3ef9b6a7c449895a90ae689ebf8dc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c8210aa708674cd58c8cc8a70d1dcf68 (127.10.238.195:38529): Not found: Tablet not found: eba7bac139fb4fafa1743db1f6661d09
W20260929 10:57:17.465826 26079 leader_election.cc:341] T eba7bac139fb4fafa1743db1f6661d09 P 7ef3ef9b6a7c449895a90ae689ebf8dc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bc5b041ccd1545fa949bed034f8066b8 (127.10.238.194:40137): Not found: Tablet not found: eba7bac139fb4fafa1743db1f6661d09
I20260929 10:57:17.465906 26079 leader_election.cc:302] T eba7bac139fb4fafa1743db1f6661d09 P 7ef3ef9b6a7c449895a90ae689ebf8dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ef3ef9b6a7c449895a90ae689ebf8dc; no voters: bc5b041ccd1545fa949bed034f8066b8, c8210aa708674cd58c8cc8a70d1dcf68
I20260929 10:57:17.465998 26474 raft_consensus.cc:2726] T eba7bac139fb4fafa1743db1f6661d09 P 7ef3ef9b6a7c449895a90ae689ebf8dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 10:57:17.883973 26475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:17.883674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.193:49553)
0929 10:57:17.883862 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:17.883904 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:40781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":107,"threads_started":1}
I20260929 10:57:17.884164 26476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:17.883913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.194:56367)
0929 10:57:17.884027 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:17.884085 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:40781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":59,"threads_started":1}
I20260929 10:57:17.953761 26477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:17.953538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.195:57097)
0929 10:57:17.953667 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:17.953705 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:40781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":61,"threads_started":1}
I20260929 10:57:18.884749 26478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:18.884523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.193:58805)
0929 10:57:18.884659 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:18.884699 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:40781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":72,"threads_started":1}
I20260929 10:57:18.884855 26479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:18.884609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.194:34097)
0929 10:57:18.884750 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:18.884783 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:40781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":91,"threads_started":1}
I20260929 10:57:18.954526 26480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:18.954288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.195:35325)
0929 10:57:18.954442 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:18.954481 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:40781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":80,"threads_started":1}
I20260929 10:57:19.885506 26481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:19.885218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.193:43793)
0929 10:57:19.885430 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:19.885460 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:40781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":144,"threads_started":1}
I20260929 10:57:19.885697 26482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:19.885394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.194:42079)
0929 10:57:19.885542 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:19.885579 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:40781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":83,"threads_started":1}
I20260929 10:57:19.955250 26483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:19.955007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.195:51513)
0929 10:57:19.955165 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:19.955207 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:40781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":87,"threads_started":1}
I20260929 10:57:20.411266 26484 raft_consensus.cc:482] T eba7bac139fb4fafa1743db1f6661d09 P 7ef3ef9b6a7c449895a90ae689ebf8dc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:57:20.411365 26484 raft_consensus.cc:504] T eba7bac139fb4fafa1743db1f6661d09 P 7ef3ef9b6a7c449895a90ae689ebf8dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8210aa708674cd58c8cc8a70d1dcf68" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38529 } } peers { permanent_uuid: "7ef3ef9b6a7c449895a90ae689ebf8dc" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 42879 } } peers { permanent_uuid: "bc5b041ccd1545fa949bed034f8066b8" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40137 } }
I20260929 10:57:20.411544 26484 leader_election.cc:288] T eba7bac139fb4fafa1743db1f6661d09 P 7ef3ef9b6a7c449895a90ae689ebf8dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c8210aa708674cd58c8cc8a70d1dcf68 (127.10.238.195:38529), bc5b041ccd1545fa949bed034f8066b8 (127.10.238.194:40137)
W20260929 10:57:20.412021 26079 leader_election.cc:341] T eba7bac139fb4fafa1743db1f6661d09 P 7ef3ef9b6a7c449895a90ae689ebf8dc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bc5b041ccd1545fa949bed034f8066b8 (127.10.238.194:40137): Not found: Tablet not found: eba7bac139fb4fafa1743db1f6661d09
W20260929 10:57:20.412104 26077 leader_election.cc:341] T eba7bac139fb4fafa1743db1f6661d09 P 7ef3ef9b6a7c449895a90ae689ebf8dc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c8210aa708674cd58c8cc8a70d1dcf68 (127.10.238.195:38529): Not found: Tablet not found: eba7bac139fb4fafa1743db1f6661d09
I20260929 10:57:20.412161 26077 leader_election.cc:302] T eba7bac139fb4fafa1743db1f6661d09 P 7ef3ef9b6a7c449895a90ae689ebf8dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ef3ef9b6a7c449895a90ae689ebf8dc; no voters: bc5b041ccd1545fa949bed034f8066b8, c8210aa708674cd58c8cc8a70d1dcf68
I20260929 10:57:20.412237 26484 raft_consensus.cc:2726] T eba7bac139fb4fafa1743db1f6661d09 P 7ef3ef9b6a7c449895a90ae689ebf8dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 10:57:20.886413 26486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:20.886205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.194:34825)
0929 10:57:20.886331 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:20.886365 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:40781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":48,"threads_started":1}
I20260929 10:57:20.886418 26485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:20.886121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.193:53651)
0929 10:57:20.886330 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:20.886371 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:40781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":115,"threads_started":1}
I20260929 10:57:20.955964 26487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:20.955715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.195:57335)
0929 10:57:20.955881 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:20.955916 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:40781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":73,"threads_started":1}
I20260929 10:57:21.887159 26488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:21.886930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.193:39923)
0929 10:57:21.887061 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:21.887101 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:40781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":58,"threads_started":1}
I20260929 10:57:21.887219 26489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:21.887015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.194:57357)
0929 10:57:21.887143 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:21.887177 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:40781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":62,"threads_started":1}
I20260929 10:57:21.956700 26490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:21.956506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.195:53255)
0929 10:57:21.956625 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:21.956658 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:40781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":58,"threads_started":1}
I20260929 10:57:22.888037 26491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:22.887772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.194:37075)
0929 10:57:22.887917 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:22.887971 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:40781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":94,"threads_started":1}
I20260929 10:57:22.888041 26492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:22.887776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.193:43569)
0929 10:57:22.887956 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:22.887994 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:40781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":90,"threads_started":1}
I20260929 10:57:22.957464 26493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:22.957221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.195:45885)
0929 10:57:22.957380 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:22.957422 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:40781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":82,"threads_started":1}
I20260929 10:57:23.261502 26494 raft_consensus.cc:482] T eba7bac139fb4fafa1743db1f6661d09 P 7ef3ef9b6a7c449895a90ae689ebf8dc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:57:23.261600 26494 raft_consensus.cc:504] T eba7bac139fb4fafa1743db1f6661d09 P 7ef3ef9b6a7c449895a90ae689ebf8dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8210aa708674cd58c8cc8a70d1dcf68" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 38529 } } peers { permanent_uuid: "7ef3ef9b6a7c449895a90ae689ebf8dc" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 42879 } } peers { permanent_uuid: "bc5b041ccd1545fa949bed034f8066b8" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 40137 } }
I20260929 10:57:23.261827 26494 leader_election.cc:288] T eba7bac139fb4fafa1743db1f6661d09 P 7ef3ef9b6a7c449895a90ae689ebf8dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c8210aa708674cd58c8cc8a70d1dcf68 (127.10.238.195:38529), bc5b041ccd1545fa949bed034f8066b8 (127.10.238.194:40137)
W20260929 10:57:23.262266 26079 leader_election.cc:341] T eba7bac139fb4fafa1743db1f6661d09 P 7ef3ef9b6a7c449895a90ae689ebf8dc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bc5b041ccd1545fa949bed034f8066b8 (127.10.238.194:40137): Not found: Tablet not found: eba7bac139fb4fafa1743db1f6661d09
W20260929 10:57:23.262391 26077 leader_election.cc:341] T eba7bac139fb4fafa1743db1f6661d09 P 7ef3ef9b6a7c449895a90ae689ebf8dc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c8210aa708674cd58c8cc8a70d1dcf68 (127.10.238.195:38529): Not found: Tablet not found: eba7bac139fb4fafa1743db1f6661d09
I20260929 10:57:23.262452 26077 leader_election.cc:302] T eba7bac139fb4fafa1743db1f6661d09 P 7ef3ef9b6a7c449895a90ae689ebf8dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ef3ef9b6a7c449895a90ae689ebf8dc; no voters: bc5b041ccd1545fa949bed034f8066b8, c8210aa708674cd58c8cc8a70d1dcf68
I20260929 10:57:23.262545 26494 raft_consensus.cc:2726] T eba7bac139fb4fafa1743db1f6661d09 P 7ef3ef9b6a7c449895a90ae689ebf8dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 10:57:23.888949 26496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:23.888697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.194:49745)
0929 10:57:23.888860 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:23.888902 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:40781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":79,"threads_started":1}
I20260929 10:57:23.888950 26495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:23.888696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.193:52223)
0929 10:57:23.888877 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:23.888910 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:40781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":103,"threads_started":1}
I20260929 10:57:23.958117 26497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:23.957902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.195:39393)
0929 10:57:23.958038 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:23.958074 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:40781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":72,"threads_started":1}
I20260929 10:57:24.889767 26498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:24.889545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.194:43937)
0929 10:57:24.889685 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:24.889719 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:40781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":77,"threads_started":1}
I20260929 10:57:24.889786 26499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:24.889545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.193:45289)
0929 10:57:24.889712 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:24.889743 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:40781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":79,"threads_started":1}
I20260929 10:57:24.958760 26500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:24.958563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.195:51907)
0929 10:57:24.958683 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:24.958717 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:40781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":59,"threads_started":1}
I20260929 10:57:25.890494 26501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:25.890247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.194:55545)
0929 10:57:25.890392 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:25.890433 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:40781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":95,"threads_started":1}
I20260929 10:57:25.890506 26502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:25.890290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.193:41863)
0929 10:57:25.890433 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:25.890467 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:40781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":64,"threads_started":1}
I20260929 10:57:25.959386 26503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:25.959197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.195:49519)
0929 10:57:25.959308 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:25.959344 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:40781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":57,"threads_started":1}
I20260929 10:57:26.891270 26504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:26.890941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.194:59959)
0929 10:57:26.891172 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:26.891222 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:40781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":55,"threads_started":1}
I20260929 10:57:26.891288 26505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:26.890952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.193:53919)
0929 10:57:26.891212 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:26.891245 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:40781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":206,"threads_started":1}
I20260929 10:57:26.959981 26506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:26.959786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.195:52117)
0929 10:57:26.959902 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:26.959935 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:40781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":61,"threads_started":1}
I20260929 10:57:27.891990 26507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:27.891777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.193:32921)
0929 10:57:27.891910 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:27.891943 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:40781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":80,"threads_started":1}
I20260929 10:57:27.892028 26508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:27.891778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.194:40543)
0929 10:57:27.891943 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:27.891980 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:40781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":93,"threads_started":1}
I20260929 10:57:27.960721 26509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:27.960488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.195:57611)
0929 10:57:27.960636 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:27.960674 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:40781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":93,"threads_started":1}
I20260929 10:57:28.892730 26511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:28.892546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.194:51607)
0929 10:57:28.892655 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:28.892689 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:40781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":68,"threads_started":1}
I20260929 10:57:28.892729 26510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:28.892493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.193:56683)
0929 10:57:28.892645 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:28.892677 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:40781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":100,"threads_started":1}
I20260929 10:57:28.961486 26512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:28.961240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:40781 (local address 127.10.238.195:36797)
0929 10:57:28.961405 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:28.961441 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:40781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":116,"threads_started":1}
I20260929 10:57:28.964900 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 26032
I20260929 10:57:28.969659 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 26196
I20260929 10:57:28.974062 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 26328
2026-09-29T10:57:28Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18439 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:57:29 dist-test-slave-8nrt krb5kdc[26515](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:57:29 dist-test-slave-8nrt krb5kdc[26515](info): set up 2 sockets
Sep 29 10:57:29 dist-test-slave-8nrt krb5kdc[26515](info): commencing operation
krb5kdc: starting...
W20260929 10:57:30.113528 11195 mini_kdc.cc:121] Time spent starting KDC: real 1.118s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Sep 29 10:57:30 dist-test-slave-8nrt krb5kdc[26515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:57:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:57:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
I20260929 10:57:30.230319 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:39121
--webserver_interface=127.10.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:34957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:39121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:57:30.310297 26528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:57:30.310503 26528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:57:30.310526 26528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:57:30.311622 26528 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 10:57:30.311654 26528 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 10:57:30.311667 26528 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 10:57:30.311678 26528 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:57:30.311690 26528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:57:30.311703 26528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:57:30.311715 26528 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:57:30.311728 26528 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:57:30.313117 26528 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:34957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:39121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:39121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:57:30.313386 26528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:57:30.313575 26528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:57:30.318943 26534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:30.319000 26533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:57:30.319015 26528 server_base.cc:1034] running on GCE node
W20260929 10:57:30.318960 26536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:57:30.319371 26528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:57:30.319703 26528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:57:30.320863 26528 hybrid_clock.cc:648] HybridClock initialized: now 1790679450320864 us; error 44 us; skew 500 ppm
Sep 29 10:57:30 dist-test-slave-8nrt krb5kdc[26515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679450, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:57:30.323416 26528 init.cc:377] Logged in from keytab as kudu/127.10.238.254@KRBTEST.COM (short username kudu)
I20260929 10:57:30.323791 26528 webserver.cc:460] Webserver started at http://127.10.238.254:44857/ using document root <none> and password file <none>
I20260929 10:57:30.323968 26528 fs_manager.cc:359] Metadata directory not provided
I20260929 10:57:30.324007 26528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:57:30.324092 26528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:57:30.325102 26528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/master-0/data/instance:
uuid: "1a72d564424948f3ac82f26f9e88be33"
format_stamp: "Formatted at 2026-09-29 10:57:30 on dist-test-slave-8nrt"
server_key: "ab9ad5a75f9560d4645d3da831e19da7"
server_key_iv: "1eca75655fecc6209f71c1ed9d455d96"
server_key_version: "encryptionkey@0"
I20260929 10:57:30.325493 26528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/master-0/wal/instance:
uuid: "1a72d564424948f3ac82f26f9e88be33"
format_stamp: "Formatted at 2026-09-29 10:57:30 on dist-test-slave-8nrt"
server_key: "ab9ad5a75f9560d4645d3da831e19da7"
server_key_iv: "1eca75655fecc6209f71c1ed9d455d96"
server_key_version: "encryptionkey@0"
I20260929 10:57:30.326711 26528 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:57:30.327363 26543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:30.327521 26528 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:57:30.327600 26528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "1a72d564424948f3ac82f26f9e88be33"
format_stamp: "Formatted at 2026-09-29 10:57:30 on dist-test-slave-8nrt"
server_key: "ab9ad5a75f9560d4645d3da831e19da7"
server_key_iv: "1eca75655fecc6209f71c1ed9d455d96"
server_key_version: "encryptionkey@0"
I20260929 10:57:30.327652 26528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:57:30.339905 26528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:57:30.341717 26528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:57:30.341825 26528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:57:30.345752 26528 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.254:39121
I20260929 10:57:30.345834 26595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.254:39121 every 8 connection(s)
I20260929 10:57:30.346160 26528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:57:30.347126 26596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:57:30.349638 26596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33: Bootstrap starting.
I20260929 10:57:30.350263 26596 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33: Neither blocks nor log segments found. Creating new log.
I20260929 10:57:30.350559 26596 log.cc:824] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33: Log is configured to *not* fsync() on all Append() calls
I20260929 10:57:30.351293 26596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33: No bootstrap required, opened a new log
I20260929 10:57:30.352982 26596 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a72d564424948f3ac82f26f9e88be33" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39121 } }
I20260929 10:57:30.353154 26596 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:57:30.353206 26596 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a72d564424948f3ac82f26f9e88be33, State: Initialized, Role: FOLLOWER
I20260929 10:57:30.353410 26596 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a72d564424948f3ac82f26f9e88be33" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39121 } }
I20260929 10:57:30.353505 26596 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:57:30.353549 26596 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:57:30.353614 26596 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:57:30.354178 26596 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a72d564424948f3ac82f26f9e88be33" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39121 } }
I20260929 10:57:30.354300 26596 leader_election.cc:302] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a72d564424948f3ac82f26f9e88be33; no voters:
I20260929 10:57:30.354492 26596 leader_election.cc:288] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:57:30.354535 26599 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:57:30.354709 26599 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33 [term 1 LEADER]: Becoming Leader. State: Replica: 1a72d564424948f3ac82f26f9e88be33, State: Running, Role: LEADER
I20260929 10:57:30.354774 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 26528
I20260929 10:57:30.354801 26596 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:57:30.354893 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/master-0/wal/instance
I20260929 10:57:30.355165 11195 external_mini_cluster.cc:1428] Setting key 81b0ff8d75bf4afe4e7717821bcbb78d
I20260929 10:57:30.354861 26599 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a72d564424948f3ac82f26f9e88be33" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39121 } }
I20260929 10:57:30.355563 26601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a72d564424948f3ac82f26f9e88be33. Latest consensus state: current_term: 1 leader_uuid: "1a72d564424948f3ac82f26f9e88be33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a72d564424948f3ac82f26f9e88be33" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39121 } } }
I20260929 10:57:30.355651 26601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33 [sys.catalog]: This master's current role is: LEADER
I20260929 10:57:30.355995 26600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a72d564424948f3ac82f26f9e88be33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a72d564424948f3ac82f26f9e88be33" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39121 } } }
I20260929 10:57:30.356068 26600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33 [sys.catalog]: This master's current role is: LEADER
I20260929 10:57:30.356226 26609 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:57:30.356741 26609 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:57:30.358264 26609 catalog_manager.cc:1347] Generated new cluster ID: cc670bf9d7ce42b9867539bb7bf6fe0b
I20260929 10:57:30.358325 26609 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 10:57:30 dist-test-slave-8nrt krb5kdc[26515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:57:30.365288 26609 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:57:30.365962 26609 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:57:30.366539 26611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:30.355861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33144 (local address 127.10.238.254:39121)
0929 10:57:30.356838 (+ 977us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:30.356848 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:30.356881 (+ 33us) server_negotiation.cc:407] Connection header received
0929 10:57:30.357001 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:30.357009 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:30.357067 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:30.357211 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:57:30.357964 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:30.358983 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:30.359746 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:30.360172 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:30.360295 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:30.362639 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:57:30.362643 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:57:30.362643 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:57:30.362653 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:57:30.364714 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:30.365288 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:30.365291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:30.365291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:30.365426 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:30.365624 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:30.365625 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:30.365626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:30.365936 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:57:30.366074 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:30.366187 (+ 113us) server_negotiation.cc:299] Negotiation successful
0929 10:57:30.366344 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":928,"thread_start_us":80,"threads_started":1}
I20260929 10:57:30.368999 26609 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33: Generated new TSK 0
I20260929 10:57:30.369160 26609 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.238.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab.
Entry for principal kudu/127.10.238.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab.
I20260929 10:57:30.396255 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.193:0
--local_ip_for_outbound_sockets=127.10.238.193
--webserver_interface=127.10.238.193
--webserver_port=0
--tserver_master_addrs=127.10.238.254:39121
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab
--principal=kudu/127.10.238.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:34957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:57:30.476619 26622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:57:30.476837 26622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:57:30.476876 26622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:57:30.478005 26622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:57:30.478067 26622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:57:30.478114 26622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.193
I20260929 10:57:30.479545 26622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:34957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab
--principal=kudu/127.10.238.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.238.193
--webserver_port=0
--tserver_master_addrs=127.10.238.254:39121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.193
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:57:30.479799 26622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:57:30.480023 26622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:57:30.484900 26627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:30.485033 26630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:57:30.484943 26622 server_base.cc:1034] running on GCE node
W20260929 10:57:30.484905 26628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:57:30.485356 26622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:57:30.485612 26622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:57:30.486776 26622 hybrid_clock.cc:648] HybridClock initialized: now 1790679450486765 us; error 33 us; skew 500 ppm
Sep 29 10:57:30 dist-test-slave-8nrt krb5kdc[26515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679450, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:57:30.489200 26622 init.cc:377] Logged in from keytab as kudu/127.10.238.193@KRBTEST.COM (short username kudu)
I20260929 10:57:30.489662 26622 webserver.cc:460] Webserver started at http://127.10.238.193:32913/ using document root <none> and password file <none>
I20260929 10:57:30.489893 26622 fs_manager.cc:359] Metadata directory not provided
I20260929 10:57:30.489938 26622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:57:30.490087 26622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:57:30.491083 26622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-0/data/instance:
uuid: "79a93325cbb74798842ddee222b050ee"
format_stamp: "Formatted at 2026-09-29 10:57:30 on dist-test-slave-8nrt"
server_key: "19c884ac07aaad4c5b3d4aeefec5a056"
server_key_iv: "76bccd5b2f16523b5cb68ed493613b48"
server_key_version: "encryptionkey@0"
I20260929 10:57:30.491418 26622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-0/wal/instance:
uuid: "79a93325cbb74798842ddee222b050ee"
format_stamp: "Formatted at 2026-09-29 10:57:30 on dist-test-slave-8nrt"
server_key: "19c884ac07aaad4c5b3d4aeefec5a056"
server_key_iv: "76bccd5b2f16523b5cb68ed493613b48"
server_key_version: "encryptionkey@0"
I20260929 10:57:30.492722 26622 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 10:57:30.493518 26637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:30.493696 26622 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:57:30.493824 26622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-0/wal
uuid: "79a93325cbb74798842ddee222b050ee"
format_stamp: "Formatted at 2026-09-29 10:57:30 on dist-test-slave-8nrt"
server_key: "19c884ac07aaad4c5b3d4aeefec5a056"
server_key_iv: "76bccd5b2f16523b5cb68ed493613b48"
server_key_version: "encryptionkey@0"
I20260929 10:57:30.493902 26622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:57:30.537302 26622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:57:30.539629 26622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:57:30.539798 26622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:57:30.540038 26622 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:57:30.540351 26622 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:57:30.540403 26622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:30.540438 26622 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:57:30.540465 26622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:30.546401 26622 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.193:32973
I20260929 10:57:30.546453 26750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.193:32973 every 8 connection(s)
I20260929 10:57:30.546754 26622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
I20260929 10:57:30.551590 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 26622
I20260929 10:57:30.551705 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-0/wal/instance
I20260929 10:57:30.551900 11195 external_mini_cluster.cc:1428] Setting key 33e2ae862d808766711760c4d4ef8a7c
Sep 29 10:57:30 dist-test-slave-8nrt krb5kdc[26515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679450, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.193@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:57:30.555987 26611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:30.547403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:60237 (local address 127.10.238.254:39121)
0929 10:57:30.547465 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:30.547467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:30.547677 (+ 210us) server_negotiation.cc:407] Connection header received
0929 10:57:30.548161 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:30.548163 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:30.548172 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:30.548212 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:57:30.549133 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:30.549651 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:30.550612 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:30.550821 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:30.550871 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:30.552955 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:57:30.552956 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:57:30.552957 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:57:30.552964 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:57:30.554492 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:30.554972 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:30.554973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:30.554974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:30.555038 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:30.555284 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:30.555285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:30.555285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:30.555405 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:57:30.555498 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:30.555801 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 10:57:30.555898 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 10:57:30.556120 26753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:30.547369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.193:60237)
0929 10:57:30.547540 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:30.547583 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:30.548064 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:30.548282 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:57:30.548286 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:57:30.548857 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:57:30.549051 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:30.549056 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:30.549734 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:30.549734 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:30.550544 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:30.550546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:30.550890 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:30.550891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:30.550989 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:30.551413 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:57:30.551438 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:57:30.552867 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:57:30.554587 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:30.554595 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:30.554597 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:30.554883 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:30.555100 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:30.555101 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:30.555101 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:30.555212 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:30.555469 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:57:30.555477 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:57:30.555619 (+ 142us) client_negotiation.cc:769] Sending connection context
0929 10:57:30.555725 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 10:57:30.555890 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":99,"threads_started":1}
I20260929 10:57:30.556769 26751 heartbeater.cc:347] Connected to a master server at 127.10.238.254:39121
I20260929 10:57:30.556849 26751 heartbeater.cc:464] Registering TS with master...
I20260929 10:57:30.557025 26751 heartbeater.cc:511] Master 127.10.238.254:39121 requested a full tablet report, sending...
I20260929 10:57:30.557578 26560 ts_manager.cc:194] Registered new tserver with Master: 79a93325cbb74798842ddee222b050ee (127.10.238.193:32973)
I20260929 10:57:30.558157 26560 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.193@KRBTEST.COM'} at 127.10.238.193:60237
WARNING: no policy specified for kudu/127.10.238.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab.
Entry for principal kudu/127.10.238.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab.
I20260929 10:57:30.576464 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.194:0
--local_ip_for_outbound_sockets=127.10.238.194
--webserver_interface=127.10.238.194
--webserver_port=0
--tserver_master_addrs=127.10.238.254:39121
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab
--principal=kudu/127.10.238.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:34957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:57:30.657421 26756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:57:30.657619 26756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:57:30.657644 26756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:57:30.658769 26756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:57:30.658811 26756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:57:30.658859 26756 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.194
I20260929 10:57:30.660378 26756 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:34957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab
--principal=kudu/127.10.238.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.238.194
--webserver_port=0
--tserver_master_addrs=127.10.238.254:39121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.194
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:57:30.660606 26756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:57:30.660827 26756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:57:30.665727 26764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:30.665838 26761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:30.665825 26762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:57:30.665905 26756 server_base.cc:1034] running on GCE node
I20260929 10:57:30.666222 26756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:57:30.666469 26756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:57:30.667627 26756 hybrid_clock.cc:648] HybridClock initialized: now 1790679450667609 us; error 35 us; skew 500 ppm
Sep 29 10:57:30 dist-test-slave-8nrt krb5kdc[26515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679450, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:57:30.670501 26756 init.cc:377] Logged in from keytab as kudu/127.10.238.194@KRBTEST.COM (short username kudu)
I20260929 10:57:30.670960 26756 webserver.cc:460] Webserver started at http://127.10.238.194:38579/ using document root <none> and password file <none>
I20260929 10:57:30.671180 26756 fs_manager.cc:359] Metadata directory not provided
I20260929 10:57:30.671244 26756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:57:30.671388 26756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:57:30.672410 26756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-1/data/instance:
uuid: "4f941f11113449088d27274866fc18d2"
format_stamp: "Formatted at 2026-09-29 10:57:30 on dist-test-slave-8nrt"
server_key: "47e10221cc0709ba7dcda1a1822f25f3"
server_key_iv: "bb5440272c32eee47cacf91cb2b36556"
server_key_version: "encryptionkey@0"
I20260929 10:57:30.672770 26756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-1/wal/instance:
uuid: "4f941f11113449088d27274866fc18d2"
format_stamp: "Formatted at 2026-09-29 10:57:30 on dist-test-slave-8nrt"
server_key: "47e10221cc0709ba7dcda1a1822f25f3"
server_key_iv: "bb5440272c32eee47cacf91cb2b36556"
server_key_version: "encryptionkey@0"
I20260929 10:57:30.674185 26756 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 10:57:30.675060 26771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:30.675431 26756 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 10:57:30.675550 26756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-1/wal
uuid: "4f941f11113449088d27274866fc18d2"
format_stamp: "Formatted at 2026-09-29 10:57:30 on dist-test-slave-8nrt"
server_key: "47e10221cc0709ba7dcda1a1822f25f3"
server_key_iv: "bb5440272c32eee47cacf91cb2b36556"
server_key_version: "encryptionkey@0"
I20260929 10:57:30.675627 26756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:57:30.690073 26756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:57:30.692147 26756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:57:30.692305 26756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:57:30.692534 26756 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:57:30.692833 26756 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:57:30.692883 26756 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:30.692924 26756 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:57:30.692950 26756 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:30.699009 26756 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.194:38929
I20260929 10:57:30.699052 26884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.194:38929 every 8 connection(s)
I20260929 10:57:30.699362 26756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
I20260929 10:57:30.700980 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 26756
I20260929 10:57:30.701102 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-1/wal/instance
I20260929 10:57:30.701406 11195 external_mini_cluster.cc:1428] Setting key 6dcb280be62d239057e78b8ba8050fd9
Sep 29 10:57:30 dist-test-slave-8nrt krb5kdc[26515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679450, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.194@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:57:30.709549 26611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:30.700074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:51379 (local address 127.10.238.254:39121)
0929 10:57:30.700171 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:30.700173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:30.700299 (+ 126us) server_negotiation.cc:407] Connection header received
0929 10:57:30.700800 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:30.700802 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:30.700817 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:30.700912 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:57:30.701981 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:30.702496 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:30.703626 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:30.703839 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:30.703907 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:30.706508 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:57:30.706509 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:57:30.706510 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:57:30.706517 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:57:30.708007 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:30.708502 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:30.708503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:30.708503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:30.708560 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:30.708850 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:30.708851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:30.708851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:30.708952 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:57:30.709007 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:30.709417 (+ 410us) server_negotiation.cc:299] Negotiation successful
0929 10:57:30.709484 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 10:57:30.709865 26887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:30.699996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.194:51379)
0929 10:57:30.700172 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:30.700213 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:30.700688 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:30.700961 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:57:30.700964 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:57:30.701672 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:57:30.701881 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:30.701886 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:30.702597 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:30.702598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:30.703529 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:30.703532 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:30.703930 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:30.703932 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:30.704083 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:30.704607 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:57:30.704617 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:57:30.706417 (+ 1800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:57:30.708091 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:30.708099 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:30.708102 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:30.708410 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:30.708633 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:30.708634 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:30.708635 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:30.708774 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:30.709019 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:57:30.709026 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:57:30.709194 (+ 168us) client_negotiation.cc:769] Sending connection context
0929 10:57:30.709391 (+ 197us) client_negotiation.cc:240] Negotiation successful
0929 10:57:30.709573 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":90,"threads_started":1}
WARNING: no policy specified for kudu/127.10.238.195@KRBTEST.COM; defaulting to no policy
I20260929 10:57:30.710426 26885 heartbeater.cc:347] Connected to a master server at 127.10.238.254:39121
I20260929 10:57:30.710523 26885 heartbeater.cc:464] Registering TS with master...
I20260929 10:57:30.710695 26885 heartbeater.cc:511] Master 127.10.238.254:39121 requested a full tablet report, sending...
I20260929 10:57:30.711054 26560 ts_manager.cc:194] Registered new tserver with Master: 4f941f11113449088d27274866fc18d2 (127.10.238.194:38929)
I20260929 10:57:30.711433 26560 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.194@KRBTEST.COM'} at 127.10.238.194:51379
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab.
Entry for principal kudu/127.10.238.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab.
I20260929 10:57:30.726235 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.195:0
--local_ip_for_outbound_sockets=127.10.238.195
--webserver_interface=127.10.238.195
--webserver_port=0
--tserver_master_addrs=127.10.238.254:39121
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab
--principal=kudu/127.10.238.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:34957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:57:30.816532 26890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:57:30.816728 26890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:57:30.816753 26890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:57:30.817992 26890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:57:30.818032 26890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:57:30.818080 26890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.195
I20260929 10:57:30.819558 26890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:34957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab
--principal=kudu/127.10.238.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.238.195
--webserver_port=0
--tserver_master_addrs=127.10.238.254:39121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.195
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:57:30.819831 26890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:57:30.820056 26890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:57:30.825230 26896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:30.825208 26898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:30.825219 26895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:57:30.825455 26890 server_base.cc:1034] running on GCE node
I20260929 10:57:30.825722 26890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:57:30.825922 26890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:57:30.827057 26890 hybrid_clock.cc:648] HybridClock initialized: now 1790679450827039 us; error 31 us; skew 500 ppm
Sep 29 10:57:30 dist-test-slave-8nrt krb5kdc[26515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679450, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:57:30.830404 26890 init.cc:377] Logged in from keytab as kudu/127.10.238.195@KRBTEST.COM (short username kudu)
I20260929 10:57:30.830952 26890 webserver.cc:460] Webserver started at http://127.10.238.195:36265/ using document root <none> and password file <none>
I20260929 10:57:30.831233 26890 fs_manager.cc:359] Metadata directory not provided
I20260929 10:57:30.831303 26890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:57:30.831437 26890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:57:30.832442 26890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-2/data/instance:
uuid: "87a7fa2b136444c389cf89f4114fb9b4"
format_stamp: "Formatted at 2026-09-29 10:57:30 on dist-test-slave-8nrt"
server_key: "9607b613a2b05d9741da6f6055369ae1"
server_key_iv: "1b9fe9c5b638cf4aa669d4766b066079"
server_key_version: "encryptionkey@0"
I20260929 10:57:30.832805 26890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-2/wal/instance:
uuid: "87a7fa2b136444c389cf89f4114fb9b4"
format_stamp: "Formatted at 2026-09-29 10:57:30 on dist-test-slave-8nrt"
server_key: "9607b613a2b05d9741da6f6055369ae1"
server_key_iv: "1b9fe9c5b638cf4aa669d4766b066079"
server_key_version: "encryptionkey@0"
I20260929 10:57:30.834256 26890 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 10:57:30.835039 26905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:30.835189 26890 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:57:30.835271 26890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-2/wal
uuid: "87a7fa2b136444c389cf89f4114fb9b4"
format_stamp: "Formatted at 2026-09-29 10:57:30 on dist-test-slave-8nrt"
server_key: "9607b613a2b05d9741da6f6055369ae1"
server_key_iv: "1b9fe9c5b638cf4aa669d4766b066079"
server_key_version: "encryptionkey@0"
I20260929 10:57:30.835326 26890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:57:30.856048 26890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:57:30.858366 26890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:57:30.858518 26890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:57:30.858736 26890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:57:30.859042 26890 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:57:30.859094 26890 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:30.859136 26890 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:57:30.859174 26890 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:30.865824 26890 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.195:36489
I20260929 10:57:30.865865 27018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.195:36489 every 8 connection(s)
I20260929 10:57:30.866225 26890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
I20260929 10:57:30.871676 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 26890
I20260929 10:57:30.871767 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-2/wal/instance
I20260929 10:57:30.871978 11195 external_mini_cluster.cc:1428] Setting key bc2d9c39889a77bd6bf0454a7f1cb0cb
Sep 29 10:57:30 dist-test-slave-8nrt krb5kdc[26515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679450, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.195@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:57:30.878738 26611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:30.866931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:37593 (local address 127.10.238.254:39121)
0929 10:57:30.867190 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:30.867192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:30.867212 (+ 20us) server_negotiation.cc:407] Connection header received
0929 10:57:30.867843 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:30.867845 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:30.867860 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:30.867950 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:57:30.869175 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:30.869948 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:30.871350 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:30.871689 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:30.871771 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:30.875150 (+ 3379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:57:30.875152 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:57:30.875153 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:57:30.875161 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:57:30.876968 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:30.877525 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:30.877526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:30.877526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:30.877593 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:30.877922 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:30.877923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:30.877923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:30.878070 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:57:30.878145 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:30.878524 (+ 379us) server_negotiation.cc:299] Negotiation successful
0929 10:57:30.878619 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228}
I20260929 10:57:30.878968 27021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:30.866929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.195:37593)
0929 10:57:30.867096 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:30.867139 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:30.867723 (+ 584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:30.868022 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:57:30.868026 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:57:30.868736 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:57:30.869041 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:30.869047 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:30.870064 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:30.870065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:30.871268 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:30.871271 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:30.871947 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:30.871949 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:30.872365 (+ 416us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:30.873137 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:57:30.873148 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:57:30.875047 (+ 1899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:57:30.877061 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:30.877068 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:30.877070 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:30.877445 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:30.877677 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:30.877678 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:30.877678 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:30.877852 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:30.878152 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:57:30.878158 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:57:30.878333 (+ 175us) client_negotiation.cc:769] Sending connection context
0929 10:57:30.878487 (+ 154us) client_negotiation.cc:240] Negotiation successful
0929 10:57:30.878698 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":101,"threads_started":1}
I20260929 10:57:30.879424 27019 heartbeater.cc:347] Connected to a master server at 127.10.238.254:39121
I20260929 10:57:30.879549 27019 heartbeater.cc:464] Registering TS with master...
I20260929 10:57:30.879760 27019 heartbeater.cc:511] Master 127.10.238.254:39121 requested a full tablet report, sending...
I20260929 10:57:30.880168 26560 ts_manager.cc:194] Registered new tserver with Master: 87a7fa2b136444c389cf89f4114fb9b4 (127.10.238.195:36489)
I20260929 10:57:30.880705 26560 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.195@KRBTEST.COM'} at 127.10.238.195:37593
I20260929 10:57:30.883638 11195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:57:30.891536 26611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:30.885077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33146 (local address 127.10.238.254:39121)
0929 10:57:30.885115 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:30.885117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:30.885157 (+ 40us) server_negotiation.cc:407] Connection header received
0929 10:57:30.885234 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:30.885234 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:30.885243 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:30.885321 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:57:30.885825 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:30.886580 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:30.887274 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:30.887597 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:30.887661 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:30.888816 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:57:30.888818 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:57:30.888818 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:57:30.888825 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:57:30.890399 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:30.890716 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:30.890717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:30.890718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:30.890781 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:30.890960 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:30.890961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:30.890961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:30.891158 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:57:30.891262 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:30.891341 (+ 79us) server_negotiation.cc:299] Negotiation successful
0929 10:57:30.891421 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 10:57:30.893069 26560 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33146:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:57:30.895665 26560 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:57:30.902992 27032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:30.898205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43298 (local address 127.10.238.195:36489)
0929 10:57:30.898395 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:30.898401 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:30.898411 (+ 10us) server_negotiation.cc:407] Connection header received
0929 10:57:30.898533 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:30.898536 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:30.898553 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:30.898610 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:57:30.898955 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:30.900085 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:30.901759 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:30.902470 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:30.902521 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:30.902572 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:30.902866 (+ 294us) server_negotiation.cc:299] Negotiation successful
0929 10:57:30.902907 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":88,"threads_started":1}
I20260929 10:57:30.902995 27033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:30.898277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46954 (local address 127.10.238.193:32973)
0929 10:57:30.898518 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:30.898522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:30.898628 (+ 106us) server_negotiation.cc:407] Connection header received
0929 10:57:30.898792 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:30.898794 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:30.898826 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:30.898906 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:57:30.899132 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:30.900214 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:30.901766 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:30.902470 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:30.902521 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:30.902572 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:30.902866 (+ 294us) server_negotiation.cc:299] Negotiation successful
0929 10:57:30.902907 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":74,"threads_started":1}
I20260929 10:57:30.903242 27034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:30.898163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:32973 (local address 127.0.0.1:46954)
0929 10:57:30.898568 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:30.898580 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:30.898650 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:30.899015 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:57:30.899016 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:57:30.899018 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:57:30.899058 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:30.899059 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:30.900320 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:30.900320 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:30.901655 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:30.901657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:30.902640 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:30.902641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:30.902738 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:30.902751 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 10:57:30.903105 (+ 354us) client_negotiation.cc:240] Negotiation successful
0929 10:57:30.903178 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":49,"thread_start_us":98,"threads_started":1}
I20260929 10:57:30.903635 27031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:30.898128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.195:36489 (local address 127.0.0.1:43298)
0929 10:57:30.898309 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:30.898349 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:30.898416 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:30.898762 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:57:30.898764 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:57:30.898766 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:57:30.898862 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:30.898865 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:30.900213 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:30.900213 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:30.901654 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:30.901657 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:30.902638 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:30.902639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:30.902738 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:30.902751 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 10:57:30.903525 (+ 774us) client_negotiation.cc:240] Negotiation successful
0929 10:57:30.903573 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":62,"thread_start_us":49,"threads_started":1}
I20260929 10:57:30.904078 27030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:30.898379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.194:38929 (local address 127.0.0.1:37814)
0929 10:57:30.899048 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:30.899059 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:30.899110 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:30.899418 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:57:30.899419 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:57:30.899422 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:57:30.899467 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:30.899469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:30.901142 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:30.901143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:30.902432 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:30.902435 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:30.903825 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:30.903826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:30.903897 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:30.903902 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 10:57:30.903951 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 10:57:30.903994 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":65,"threads_started":1}
I20260929 10:57:30.904002 26685 tablet_service.cc:1467] Processing CreateTablet for tablet 6b61f157d65849f1b2905825497b0dae (DEFAULT_TABLE table=test-table [id=2daa22959cdd47abaf2a0228d2f0d91e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:57:30.904358 26685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b61f157d65849f1b2905825497b0dae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:57:30.904407 27035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:30.898519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37814 (local address 127.10.238.194:38929)
0929 10:57:30.898685 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:30.898692 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:30.899176 (+ 484us) server_negotiation.cc:407] Connection header received
0929 10:57:30.899227 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:30.899229 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:30.899247 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:30.899326 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:57:30.899539 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:30.901034 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:30.902538 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:30.903665 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:30.903739 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:30.903789 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:30.904245 (+ 456us) server_negotiation.cc:299] Negotiation successful
0929 10:57:30.904307 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":83,"threads_started":1}
I20260929 10:57:30.904474 26953 tablet_service.cc:1467] Processing CreateTablet for tablet 6b61f157d65849f1b2905825497b0dae (DEFAULT_TABLE table=test-table [id=2daa22959cdd47abaf2a0228d2f0d91e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:57:30.904755 26953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b61f157d65849f1b2905825497b0dae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:57:30.904937 26819 tablet_service.cc:1467] Processing CreateTablet for tablet 6b61f157d65849f1b2905825497b0dae (DEFAULT_TABLE table=test-table [id=2daa22959cdd47abaf2a0228d2f0d91e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:57:30.905164 26819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b61f157d65849f1b2905825497b0dae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:57:30.907724 27036 tablet_bootstrap.cc:492] T 6b61f157d65849f1b2905825497b0dae P 4f941f11113449088d27274866fc18d2: Bootstrap starting.
I20260929 10:57:30.908044 27037 tablet_bootstrap.cc:492] T 6b61f157d65849f1b2905825497b0dae P 79a93325cbb74798842ddee222b050ee: Bootstrap starting.
I20260929 10:57:30.908421 27036 tablet_bootstrap.cc:654] T 6b61f157d65849f1b2905825497b0dae P 4f941f11113449088d27274866fc18d2: Neither blocks nor log segments found. Creating new log.
I20260929 10:57:30.908583 27038 tablet_bootstrap.cc:492] T 6b61f157d65849f1b2905825497b0dae P 87a7fa2b136444c389cf89f4114fb9b4: Bootstrap starting.
I20260929 10:57:30.908713 27037 tablet_bootstrap.cc:654] T 6b61f157d65849f1b2905825497b0dae P 79a93325cbb74798842ddee222b050ee: Neither blocks nor log segments found. Creating new log.
I20260929 10:57:30.908998 27036 log.cc:824] T 6b61f157d65849f1b2905825497b0dae P 4f941f11113449088d27274866fc18d2: Log is configured to *not* fsync() on all Append() calls
I20260929 10:57:30.909348 27037 log.cc:824] T 6b61f157d65849f1b2905825497b0dae P 79a93325cbb74798842ddee222b050ee: Log is configured to *not* fsync() on all Append() calls
I20260929 10:57:30.909512 27038 tablet_bootstrap.cc:654] T 6b61f157d65849f1b2905825497b0dae P 87a7fa2b136444c389cf89f4114fb9b4: Neither blocks nor log segments found. Creating new log.
I20260929 10:57:30.909988 27036 tablet_bootstrap.cc:492] T 6b61f157d65849f1b2905825497b0dae P 4f941f11113449088d27274866fc18d2: No bootstrap required, opened a new log
I20260929 10:57:30.910012 27038 log.cc:824] T 6b61f157d65849f1b2905825497b0dae P 87a7fa2b136444c389cf89f4114fb9b4: Log is configured to *not* fsync() on all Append() calls
I20260929 10:57:30.910064 27036 ts_tablet_manager.cc:1397] T 6b61f157d65849f1b2905825497b0dae P 4f941f11113449088d27274866fc18d2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 10:57:30.910665 27038 tablet_bootstrap.cc:492] T 6b61f157d65849f1b2905825497b0dae P 87a7fa2b136444c389cf89f4114fb9b4: No bootstrap required, opened a new log
I20260929 10:57:30.910740 27038 ts_tablet_manager.cc:1397] T 6b61f157d65849f1b2905825497b0dae P 87a7fa2b136444c389cf89f4114fb9b4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 10:57:30.912015 27038 raft_consensus.cc:348] T 6b61f157d65849f1b2905825497b0dae P 87a7fa2b136444c389cf89f4114fb9b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87a7fa2b136444c389cf89f4114fb9b4" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36489 } } peers { permanent_uuid: "79a93325cbb74798842ddee222b050ee" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 32973 } } peers { permanent_uuid: "4f941f11113449088d27274866fc18d2" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38929 } }
I20260929 10:57:30.912168 27038 raft_consensus.cc:374] T 6b61f157d65849f1b2905825497b0dae P 87a7fa2b136444c389cf89f4114fb9b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:57:30.912221 27038 raft_consensus.cc:729] T 6b61f157d65849f1b2905825497b0dae P 87a7fa2b136444c389cf89f4114fb9b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87a7fa2b136444c389cf89f4114fb9b4, State: Initialized, Role: FOLLOWER
I20260929 10:57:30.912333 27038 consensus_queue.cc:260] T 6b61f157d65849f1b2905825497b0dae P 87a7fa2b136444c389cf89f4114fb9b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87a7fa2b136444c389cf89f4114fb9b4" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36489 } } peers { permanent_uuid: "79a93325cbb74798842ddee222b050ee" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 32973 } } peers { permanent_uuid: "4f941f11113449088d27274866fc18d2" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38929 } }
I20260929 10:57:30.912310 27036 raft_consensus.cc:348] T 6b61f157d65849f1b2905825497b0dae P 4f941f11113449088d27274866fc18d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87a7fa2b136444c389cf89f4114fb9b4" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36489 } } peers { permanent_uuid: "79a93325cbb74798842ddee222b050ee" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 32973 } } peers { permanent_uuid: "4f941f11113449088d27274866fc18d2" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38929 } }
I20260929 10:57:30.912489 27036 raft_consensus.cc:374] T 6b61f157d65849f1b2905825497b0dae P 4f941f11113449088d27274866fc18d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:57:30.912532 27036 raft_consensus.cc:729] T 6b61f157d65849f1b2905825497b0dae P 4f941f11113449088d27274866fc18d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f941f11113449088d27274866fc18d2, State: Initialized, Role: FOLLOWER
I20260929 10:57:30.912613 27038 ts_tablet_manager.cc:1428] T 6b61f157d65849f1b2905825497b0dae P 87a7fa2b136444c389cf89f4114fb9b4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 10:57:30.912635 27019 heartbeater.cc:503] Master 127.10.238.254:39121 was elected leader, sending a full tablet report...
I20260929 10:57:30.912655 27036 consensus_queue.cc:260] T 6b61f157d65849f1b2905825497b0dae P 4f941f11113449088d27274866fc18d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87a7fa2b136444c389cf89f4114fb9b4" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36489 } } peers { permanent_uuid: "79a93325cbb74798842ddee222b050ee" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 32973 } } peers { permanent_uuid: "4f941f11113449088d27274866fc18d2" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38929 } }
I20260929 10:57:30.912935 27036 ts_tablet_manager.cc:1428] T 6b61f157d65849f1b2905825497b0dae P 4f941f11113449088d27274866fc18d2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 10:57:30.912973 26885 heartbeater.cc:503] Master 127.10.238.254:39121 was elected leader, sending a full tablet report...
I20260929 10:57:30.913522 27037 tablet_bootstrap.cc:492] T 6b61f157d65849f1b2905825497b0dae P 79a93325cbb74798842ddee222b050ee: No bootstrap required, opened a new log
I20260929 10:57:30.913594 27037 ts_tablet_manager.cc:1397] T 6b61f157d65849f1b2905825497b0dae P 79a93325cbb74798842ddee222b050ee: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260929 10:57:30.915515 27037 raft_consensus.cc:348] T 6b61f157d65849f1b2905825497b0dae P 79a93325cbb74798842ddee222b050ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87a7fa2b136444c389cf89f4114fb9b4" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36489 } } peers { permanent_uuid: "79a93325cbb74798842ddee222b050ee" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 32973 } } peers { permanent_uuid: "4f941f11113449088d27274866fc18d2" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38929 } }
I20260929 10:57:30.915640 27037 raft_consensus.cc:374] T 6b61f157d65849f1b2905825497b0dae P 79a93325cbb74798842ddee222b050ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:57:30.915663 27037 raft_consensus.cc:729] T 6b61f157d65849f1b2905825497b0dae P 79a93325cbb74798842ddee222b050ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79a93325cbb74798842ddee222b050ee, State: Initialized, Role: FOLLOWER
I20260929 10:57:30.915804 27037 consensus_queue.cc:260] T 6b61f157d65849f1b2905825497b0dae P 79a93325cbb74798842ddee222b050ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87a7fa2b136444c389cf89f4114fb9b4" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36489 } } peers { permanent_uuid: "79a93325cbb74798842ddee222b050ee" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 32973 } } peers { permanent_uuid: "4f941f11113449088d27274866fc18d2" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38929 } }
I20260929 10:57:30.916080 27037 ts_tablet_manager.cc:1428] T 6b61f157d65849f1b2905825497b0dae P 79a93325cbb74798842ddee222b050ee: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260929 10:57:30.916141 26751 heartbeater.cc:503] Master 127.10.238.254:39121 was elected leader, sending a full tablet report...
W20260929 10:57:30.949977 26886 tablet.cc:2369] T 6b61f157d65849f1b2905825497b0dae P 4f941f11113449088d27274866fc18d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 10:57:31.047442 26752 tablet.cc:2369] T 6b61f157d65849f1b2905825497b0dae P 79a93325cbb74798842ddee222b050ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 10:57:31.117200 27020 tablet.cc:2369] T 6b61f157d65849f1b2905825497b0dae P 87a7fa2b136444c389cf89f4114fb9b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:57:31.121464 27043 raft_consensus.cc:482] T 6b61f157d65849f1b2905825497b0dae P 4f941f11113449088d27274866fc18d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:57:31.121600 27043 raft_consensus.cc:504] T 6b61f157d65849f1b2905825497b0dae P 4f941f11113449088d27274866fc18d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87a7fa2b136444c389cf89f4114fb9b4" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36489 } } peers { permanent_uuid: "79a93325cbb74798842ddee222b050ee" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 32973 } } peers { permanent_uuid: "4f941f11113449088d27274866fc18d2" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38929 } }
I20260929 10:57:31.122025 27043 leader_election.cc:288] T 6b61f157d65849f1b2905825497b0dae P 4f941f11113449088d27274866fc18d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87a7fa2b136444c389cf89f4114fb9b4 (127.10.238.195:36489), 79a93325cbb74798842ddee222b050ee (127.10.238.193:32973)
I20260929 10:57:31.124905 27032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:31.122363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:42171 (local address 127.10.238.195:36489)
0929 10:57:31.122417 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:31.122420 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:31.122430 (+ 10us) server_negotiation.cc:407] Connection header received
0929 10:57:31.122456 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:31.122458 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:31.122468 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:31.122521 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:57:31.122719 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:31.123205 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:31.124142 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:31.124520 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:31.124613 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:31.124623 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:31.124790 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 10:57:31.124838 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 10:57:31.124924 26887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:31.122206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.195:36489 (local address 127.10.238.194:42171)
0929 10:57:31.122324 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:31.122337 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:31.122384 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:31.122553 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:57:31.122555 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:57:31.122557 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:57:31.122642 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:31.122644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:31.123294 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:31.123296 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:31.124052 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:31.124054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:31.124613 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:31.124614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:31.124701 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:31.124704 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 10:57:31.124774 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 10:57:31.124820 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":19}
I20260929 10:57:31.124914 27033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:31.122438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:49639 (local address 127.10.238.193:32973)
0929 10:57:31.122486 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:31.122488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:31.122497 (+ 9us) server_negotiation.cc:407] Connection header received
0929 10:57:31.122526 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:31.122527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:31.122536 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:31.122584 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:57:31.122785 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:31.123249 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:31.124218 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:31.124564 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:31.124641 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:31.124649 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:31.124806 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 10:57:31.124845 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 10:57:31.124948 27045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:31.122206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:32973 (local address 127.10.238.194:49639)
0929 10:57:31.122404 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:31.122418 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:31.122475 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:31.122626 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:57:31.122627 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:57:31.122629 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:57:31.122711 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:31.122713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:31.123330 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:31.123331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:31.124103 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:31.124105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:31.124641 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:31.124642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:31.124727 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:31.124730 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 10:57:31.124796 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 10:57:31.124833 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":15,"thread_start_us":111,"threads_started":1}
I20260929 10:57:31.125303 26973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b61f157d65849f1b2905825497b0dae" candidate_uuid: "4f941f11113449088d27274866fc18d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87a7fa2b136444c389cf89f4114fb9b4" is_pre_election: true
I20260929 10:57:31.125463 26973 raft_consensus.cc:2454] T 6b61f157d65849f1b2905825497b0dae P 87a7fa2b136444c389cf89f4114fb9b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f941f11113449088d27274866fc18d2 in term 0.
I20260929 10:57:31.125732 26773 leader_election.cc:302] T 6b61f157d65849f1b2905825497b0dae P 4f941f11113449088d27274866fc18d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f941f11113449088d27274866fc18d2, 87a7fa2b136444c389cf89f4114fb9b4; no voters:
I20260929 10:57:31.125787 26705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b61f157d65849f1b2905825497b0dae" candidate_uuid: "4f941f11113449088d27274866fc18d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79a93325cbb74798842ddee222b050ee" is_pre_election: true
I20260929 10:57:31.125885 27043 raft_consensus.cc:2781] T 6b61f157d65849f1b2905825497b0dae P 4f941f11113449088d27274866fc18d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:57:31.125936 27043 raft_consensus.cc:482] T 6b61f157d65849f1b2905825497b0dae P 4f941f11113449088d27274866fc18d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:57:31.125936 26705 raft_consensus.cc:2454] T 6b61f157d65849f1b2905825497b0dae P 79a93325cbb74798842ddee222b050ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f941f11113449088d27274866fc18d2 in term 0.
I20260929 10:57:31.125969 27043 raft_consensus.cc:3037] T 6b61f157d65849f1b2905825497b0dae P 4f941f11113449088d27274866fc18d2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:57:31.126732 27043 raft_consensus.cc:504] T 6b61f157d65849f1b2905825497b0dae P 4f941f11113449088d27274866fc18d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87a7fa2b136444c389cf89f4114fb9b4" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36489 } } peers { permanent_uuid: "79a93325cbb74798842ddee222b050ee" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 32973 } } peers { permanent_uuid: "4f941f11113449088d27274866fc18d2" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38929 } }
I20260929 10:57:31.126878 27043 leader_election.cc:288] T 6b61f157d65849f1b2905825497b0dae P 4f941f11113449088d27274866fc18d2 [CANDIDATE]: Term 1 election: Requested vote from peers 87a7fa2b136444c389cf89f4114fb9b4 (127.10.238.195:36489), 79a93325cbb74798842ddee222b050ee (127.10.238.193:32973)
I20260929 10:57:31.127041 26705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b61f157d65849f1b2905825497b0dae" candidate_uuid: "4f941f11113449088d27274866fc18d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79a93325cbb74798842ddee222b050ee"
I20260929 10:57:31.127075 26973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b61f157d65849f1b2905825497b0dae" candidate_uuid: "4f941f11113449088d27274866fc18d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87a7fa2b136444c389cf89f4114fb9b4"
I20260929 10:57:31.127139 26973 raft_consensus.cc:3037] T 6b61f157d65849f1b2905825497b0dae P 87a7fa2b136444c389cf89f4114fb9b4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:57:31.127138 26705 raft_consensus.cc:3037] T 6b61f157d65849f1b2905825497b0dae P 79a93325cbb74798842ddee222b050ee [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:57:31.127852 26705 raft_consensus.cc:2454] T 6b61f157d65849f1b2905825497b0dae P 79a93325cbb74798842ddee222b050ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f941f11113449088d27274866fc18d2 in term 1.
I20260929 10:57:31.127964 26973 raft_consensus.cc:2454] T 6b61f157d65849f1b2905825497b0dae P 87a7fa2b136444c389cf89f4114fb9b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f941f11113449088d27274866fc18d2 in term 1.
I20260929 10:57:31.128067 26772 leader_election.cc:302] T 6b61f157d65849f1b2905825497b0dae P 4f941f11113449088d27274866fc18d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f941f11113449088d27274866fc18d2, 79a93325cbb74798842ddee222b050ee; no voters:
I20260929 10:57:31.128161 27043 raft_consensus.cc:2781] T 6b61f157d65849f1b2905825497b0dae P 4f941f11113449088d27274866fc18d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:57:31.128314 27043 raft_consensus.cc:686] T 6b61f157d65849f1b2905825497b0dae P 4f941f11113449088d27274866fc18d2 [term 1 LEADER]: Becoming Leader. State: Replica: 4f941f11113449088d27274866fc18d2, State: Running, Role: LEADER
I20260929 10:57:31.128417 27043 consensus_queue.cc:237] T 6b61f157d65849f1b2905825497b0dae P 4f941f11113449088d27274866fc18d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87a7fa2b136444c389cf89f4114fb9b4" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36489 } } peers { permanent_uuid: "79a93325cbb74798842ddee222b050ee" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 32973 } } peers { permanent_uuid: "4f941f11113449088d27274866fc18d2" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38929 } }
I20260929 10:57:31.129321 26559 catalog_manager.cc:5495] T 6b61f157d65849f1b2905825497b0dae P 4f941f11113449088d27274866fc18d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f941f11113449088d27274866fc18d2 (127.10.238.194). New cstate: current_term: 1 leader_uuid: "4f941f11113449088d27274866fc18d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87a7fa2b136444c389cf89f4114fb9b4" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79a93325cbb74798842ddee222b050ee" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 32973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f941f11113449088d27274866fc18d2" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38929 } health_report { overall_health: HEALTHY } } }
I20260929 10:57:31.183399 27035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:31.181010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37822 (local address 127.10.238.194:38929)
0929 10:57:31.181090 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:31.181092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:31.181102 (+ 10us) server_negotiation.cc:407] Connection header received
0929 10:57:31.181154 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:31.181155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:31.181174 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:31.181249 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:57:31.181391 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:31.181862 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:31.182622 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:31.182823 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:31.182897 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:31.183054 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:57:31.183165 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:57:31.183259 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:31.183286 (+ 27us) server_negotiation.cc:299] Negotiation successful
0929 10:57:31.183307 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 10:57:31.185300 26973 raft_consensus.cc:1261] T 6b61f157d65849f1b2905825497b0dae P 87a7fa2b136444c389cf89f4114fb9b4 [term 1 FOLLOWER]: Refusing update from remote peer 4f941f11113449088d27274866fc18d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:57:31.185323 26705 raft_consensus.cc:1261] T 6b61f157d65849f1b2905825497b0dae P 79a93325cbb74798842ddee222b050ee [term 1 FOLLOWER]: Refusing update from remote peer 4f941f11113449088d27274866fc18d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:57:31.185689 27043 consensus_queue.cc:1035] T 6b61f157d65849f1b2905825497b0dae P 4f941f11113449088d27274866fc18d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87a7fa2b136444c389cf89f4114fb9b4" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:57:31.185847 27046 consensus_queue.cc:1035] T 6b61f157d65849f1b2905825497b0dae P 4f941f11113449088d27274866fc18d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79a93325cbb74798842ddee222b050ee" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 32973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:57:31.191123 26559 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:57:31.191217 26559 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:57:31.191998 26559 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33: Sending DeleteTablet for 3 replicas of tablet 6b61f157d65849f1b2905825497b0dae
I20260929 10:57:31.192337 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 26622
I20260929 10:57:31.192306 26953 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b61f157d65849f1b2905825497b0dae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:57:31 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:43298
I20260929 10:57:31.192443 26685 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b61f157d65849f1b2905825497b0dae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:57:31 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:46954
I20260929 10:57:31.192860 26819 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b61f157d65849f1b2905825497b0dae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:57:31 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:37814
I20260929 10:57:31.193078 27060 tablet_replica.cc:330] T 6b61f157d65849f1b2905825497b0dae P 4f941f11113449088d27274866fc18d2: stopping tablet replica
I20260929 10:57:31.193216 27060 raft_consensus.cc:2229] T 6b61f157d65849f1b2905825497b0dae P 4f941f11113449088d27274866fc18d2 [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:57:31.193411 27060 raft_consensus.cc:2258] T 6b61f157d65849f1b2905825497b0dae P 4f941f11113449088d27274866fc18d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:57:31.194165 27059 tablet_replica.cc:330] T 6b61f157d65849f1b2905825497b0dae P 87a7fa2b136444c389cf89f4114fb9b4: stopping tablet replica
I20260929 10:57:31.194293 27059 raft_consensus.cc:2229] T 6b61f157d65849f1b2905825497b0dae P 87a7fa2b136444c389cf89f4114fb9b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:57:31.194404 27059 raft_consensus.cc:2258] T 6b61f157d65849f1b2905825497b0dae P 87a7fa2b136444c389cf89f4114fb9b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:57:31.194856 27059 ts_tablet_manager.cc:1905] T 6b61f157d65849f1b2905825497b0dae P 87a7fa2b136444c389cf89f4114fb9b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:57:31.196892 27059 ts_tablet_manager.cc:1918] T 6b61f157d65849f1b2905825497b0dae P 87a7fa2b136444c389cf89f4114fb9b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:57:31.196971 27059 log.cc:1196] T 6b61f157d65849f1b2905825497b0dae P 87a7fa2b136444c389cf89f4114fb9b4: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-2/wal/wals/6b61f157d65849f1b2905825497b0dae
I20260929 10:57:31.197322 27059 ts_tablet_manager.cc:1939] T 6b61f157d65849f1b2905825497b0dae P 87a7fa2b136444c389cf89f4114fb9b4: Deleting consensus metadata
I20260929 10:57:31.197749 26545 catalog_manager.cc:4841] TS 87a7fa2b136444c389cf89f4114fb9b4 (127.10.238.195:36489): tablet 6b61f157d65849f1b2905825497b0dae (table test-table [id=2daa22959cdd47abaf2a0228d2f0d91e]) successfully deleted
I20260929 10:57:31.198377 27060 ts_tablet_manager.cc:1905] T 6b61f157d65849f1b2905825497b0dae P 4f941f11113449088d27274866fc18d2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 10:57:31.199303 26544 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.238.193:32973 (error 108)
I20260929 10:57:31.199584 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 26756
I20260929 10:57:31.200590 27031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:31.200453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:32973 (local address 127.0.0.1:46966)
0929 10:57:31.200500 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:31.200554 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.193:32973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260929 10:57:31.200920 26544 catalog_manager.cc:4584] TS 79a93325cbb74798842ddee222b050ee (127.10.238.193:32973): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6b61f157d65849f1b2905825497b0dae: Network error: Client connection negotiation failed: client connection to 127.10.238.193:32973: connect: Connection refused (error 111)
I20260929 10:57:31.204249 27031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:31.204129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.194:38929 (local address 127.0.0.1:37836)
0929 10:57:31.204184 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:31.204212 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.194:38929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 10:57:31.204321 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 26890
I20260929 10:57:31.209033 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 26528
I20260929 10:57:51.214493 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:39121
--webserver_interface=127.10.238.254
--webserver_port=44857
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:34957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:39121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:57:51.298935 27062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:57:51.299214 27062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:57:51.299258 27062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:57:51.300472 27062 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 10:57:51.300547 27062 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 10:57:51.300575 27062 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 10:57:51.300614 27062 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:57:51.300652 27062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:57:51.300683 27062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:57:51.300717 27062 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:57:51.300751 27062 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:57:51.302335 27062 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:34957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:39121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:39121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.238.254
--webserver_port=44857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:57:51.302654 27062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:57:51.302897 27062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:57:51.312845 27071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:51.312961 27069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:51.312963 27068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:57:51.312994 27062 server_base.cc:1034] running on GCE node
I20260929 10:57:51.313417 27062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:57:51.313761 27062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:57:51.314947 27062 hybrid_clock.cc:648] HybridClock initialized: now 1790679471314936 us; error 31 us; skew 500 ppm
Sep 29 10:57:51 dist-test-slave-8nrt krb5kdc[26515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679471, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:57:51.320742 27062 init.cc:377] Logged in from keytab as kudu/127.10.238.254@KRBTEST.COM (short username kudu)
I20260929 10:57:51.321229 27062 webserver.cc:460] Webserver started at http://127.10.238.254:44857/ using document root <none> and password file <none>
I20260929 10:57:51.321538 27062 fs_manager.cc:359] Metadata directory not provided
I20260929 10:57:51.321627 27062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:57:51.323048 27062 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 10:57:51.323860 27078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:51.324131 27062 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:57:51.324221 27062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "1a72d564424948f3ac82f26f9e88be33"
format_stamp: "Formatted at 2026-09-29 10:57:30 on dist-test-slave-8nrt"
server_key: "ab9ad5a75f9560d4645d3da831e19da7"
server_key_iv: "1eca75655fecc6209f71c1ed9d455d96"
server_key_version: "encryptionkey@0"
I20260929 10:57:51.324723 27062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:57:51.338208 27062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:57:51.340214 27062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:57:51.340343 27062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:57:51.344391 27062 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.254:39121
I20260929 10:57:51.344506 27130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.254:39121 every 8 connection(s)
I20260929 10:57:51.344882 27062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:57:51.346525 27131 sys_catalog.cc:263] Verifying existing consensus state
I20260929 10:57:51.347208 27131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33: Bootstrap starting.
I20260929 10:57:51.350083 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 27062
I20260929 10:57:51.350385 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.193:32973
--local_ip_for_outbound_sockets=127.10.238.193
--tserver_master_addrs=127.10.238.254:39121
--webserver_port=32913
--webserver_interface=127.10.238.193
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab
--principal=kudu/127.10.238.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:34957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:57:51.356532 27131 log.cc:824] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33: Log is configured to *not* fsync() on all Append() calls
I20260929 10:57:51.358593 27131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 10:57:51.358928 27131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33: Bootstrap complete.
I20260929 10:57:51.361302 27131 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a72d564424948f3ac82f26f9e88be33" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39121 } }
I20260929 10:57:51.361585 27131 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a72d564424948f3ac82f26f9e88be33, State: Initialized, Role: FOLLOWER
I20260929 10:57:51.361802 27131 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a72d564424948f3ac82f26f9e88be33" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39121 } }
I20260929 10:57:51.361899 27131 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:57:51.361960 27131 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:57:51.362008 27131 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33 [term 1 FOLLOWER]: Advancing to term 2
I20260929 10:57:51.363070 27131 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a72d564424948f3ac82f26f9e88be33" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39121 } }
I20260929 10:57:51.363200 27131 leader_election.cc:302] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a72d564424948f3ac82f26f9e88be33; no voters:
I20260929 10:57:51.363405 27131 leader_election.cc:288] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 10:57:51.363445 27135 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 10:57:51.363564 27135 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33 [term 2 LEADER]: Becoming Leader. State: Replica: 1a72d564424948f3ac82f26f9e88be33, State: Running, Role: LEADER
I20260929 10:57:51.363651 27135 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a72d564424948f3ac82f26f9e88be33" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39121 } }
I20260929 10:57:51.363701 27131 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:57:51.364028 27135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a72d564424948f3ac82f26f9e88be33. Latest consensus state: current_term: 2 leader_uuid: "1a72d564424948f3ac82f26f9e88be33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a72d564424948f3ac82f26f9e88be33" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39121 } } }
I20260929 10:57:51.364104 27135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33 [sys.catalog]: This master's current role is: LEADER
I20260929 10:57:51.364059 27136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1a72d564424948f3ac82f26f9e88be33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a72d564424948f3ac82f26f9e88be33" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39121 } } }
I20260929 10:57:51.364341 27141 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:57:51.364369 27136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33 [sys.catalog]: This master's current role is: LEADER
I20260929 10:57:51.365208 27141 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:57:51.365338 27141 catalog_manager.cc:1259] Loaded cluster ID: cc670bf9d7ce42b9867539bb7bf6fe0b
I20260929 10:57:51.365378 27141 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:57:51.366245 27141 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:57:51.366425 27141 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33: Loaded TSK: 0
I20260929 10:57:51.377585 27141 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33: Generated new TSK 1
I20260929 10:57:51.377712 27141 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 10:57:51.443423 27133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:57:51.443624 27133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:57:51.443648 27133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:57:51.444816 27133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:57:51.444857 27133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:57:51.444885 27133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.193
I20260929 10:57:51.446405 27133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:34957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab
--principal=kudu/127.10.238.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.193:32973
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.238.193
--webserver_port=32913
--tserver_master_addrs=127.10.238.254:39121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.193
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:57:51.446640 27133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:57:51.446919 27133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:57:51.451911 27159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:51.451922 27158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:51.451922 27161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:57:51.452292 27133 server_base.cc:1034] running on GCE node
I20260929 10:57:51.452576 27133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:57:51.452858 27133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:57:51.454034 27133 hybrid_clock.cc:648] HybridClock initialized: now 1790679471454011 us; error 41 us; skew 500 ppm
Sep 29 10:57:51 dist-test-slave-8nrt krb5kdc[26515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679471, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:57:51.456614 27133 init.cc:377] Logged in from keytab as kudu/127.10.238.193@KRBTEST.COM (short username kudu)
I20260929 10:57:51.457069 27133 webserver.cc:460] Webserver started at http://127.10.238.193:32913/ using document root <none> and password file <none>
I20260929 10:57:51.457362 27133 fs_manager.cc:359] Metadata directory not provided
I20260929 10:57:51.457420 27133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:57:51.458885 27133 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:57:51.459667 27168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:51.459882 27133 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:57:51.459983 27133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-0/wal
uuid: "79a93325cbb74798842ddee222b050ee"
format_stamp: "Formatted at 2026-09-29 10:57:30 on dist-test-slave-8nrt"
server_key: "19c884ac07aaad4c5b3d4aeefec5a056"
server_key_iv: "76bccd5b2f16523b5cb68ed493613b48"
server_key_version: "encryptionkey@0"
I20260929 10:57:51.460371 27133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:57:51.488399 27133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:57:51.490749 27133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:57:51.490926 27133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:57:51.491176 27133 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:57:51.491667 27175 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 10:57:51.493531 27133 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 10:57:51.493595 27133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260929 10:57:51.493642 27133 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 10:57:51.494380 27133 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 10:57:51.494434 27133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260929 10:57:51.494503 27175 tablet_bootstrap.cc:492] T 6b61f157d65849f1b2905825497b0dae P 79a93325cbb74798842ddee222b050ee: Bootstrap starting.
I20260929 10:57:51.501689 27133 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.193:32973
I20260929 10:57:51.501842 27282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.193:32973 every 8 connection(s)
I20260929 10:57:51.502125 27133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
I20260929 10:57:51.505987 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 27133
I20260929 10:57:51.506287 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.194:38929
--local_ip_for_outbound_sockets=127.10.238.194
--tserver_master_addrs=127.10.238.254:39121
--webserver_port=38579
--webserver_interface=127.10.238.194
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab
--principal=kudu/127.10.238.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:34957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:57:51.509083 27175 log.cc:824] T 6b61f157d65849f1b2905825497b0dae P 79a93325cbb74798842ddee222b050ee: Log is configured to *not* fsync() on all Append() calls
Sep 29 10:57:51 dist-test-slave-8nrt krb5kdc[26515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679471, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.193@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:57:51.511952 27175 tablet_bootstrap.cc:492] T 6b61f157d65849f1b2905825497b0dae P 79a93325cbb74798842ddee222b050ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 10:57:51.512307 27175 tablet_bootstrap.cc:492] T 6b61f157d65849f1b2905825497b0dae P 79a93325cbb74798842ddee222b050ee: Bootstrap complete.
I20260929 10:57:51.513654 27286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:51.503116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:52589 (local address 127.10.238.254:39121)
0929 10:57:51.503299 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:51.503308 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:51.503359 (+ 51us) server_negotiation.cc:407] Connection header received
0929 10:57:51.503864 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:51.503869 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:51.503912 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:51.504057 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:57:51.504671 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:51.505681 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:51.506902 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:51.507290 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:51.507404 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:51.510059 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:57:51.510063 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:57:51.510063 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:57:51.510075 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:57:51.511791 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:51.512318 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:51.512321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:51.512321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:51.512434 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:51.512714 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:51.512715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:51.512716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:51.512933 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:57:51.513019 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:51.513342 (+ 323us) server_negotiation.cc:299] Negotiation successful
0929 10:57:51.513467 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":78,"threads_started":1}
I20260929 10:57:51.513666 27285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:51.502851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.193:52589)
0929 10:57:51.503017 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:51.503053 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:51.503587 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:51.504087 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:57:51.504091 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:57:51.504366 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:57:51.504556 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:51.504560 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:51.505826 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:51.505827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:51.506808 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:51.506811 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:51.507407 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:51.507408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:51.507534 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:51.508120 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:57:51.508131 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:57:51.509942 (+ 1811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:57:51.511896 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:51.511903 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:51.511905 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:51.512220 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:51.512521 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:51.512522 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:51.512522 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:51.512633 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:51.513011 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:57:51.513018 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:57:51.513151 (+ 133us) client_negotiation.cc:769] Sending connection context
0929 10:57:51.513264 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 10:57:51.513467 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":95,"threads_started":1}
I20260929 10:57:51.513728 27175 ts_tablet_manager.cc:1397] T 6b61f157d65849f1b2905825497b0dae P 79a93325cbb74798842ddee222b050ee: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.012s
I20260929 10:57:51.514766 27283 heartbeater.cc:347] Connected to a master server at 127.10.238.254:39121
I20260929 10:57:51.514854 27283 heartbeater.cc:464] Registering TS with master...
I20260929 10:57:51.515021 27283 heartbeater.cc:511] Master 127.10.238.254:39121 requested a full tablet report, sending...
I20260929 10:57:51.515429 27175 raft_consensus.cc:348] T 6b61f157d65849f1b2905825497b0dae P 79a93325cbb74798842ddee222b050ee [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87a7fa2b136444c389cf89f4114fb9b4" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36489 } } peers { permanent_uuid: "79a93325cbb74798842ddee222b050ee" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 32973 } } peers { permanent_uuid: "4f941f11113449088d27274866fc18d2" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38929 } }
I20260929 10:57:51.515741 27095 ts_manager.cc:194] Registered new tserver with Master: 79a93325cbb74798842ddee222b050ee (127.10.238.193:32973)
I20260929 10:57:51.515753 27175 raft_consensus.cc:729] T 6b61f157d65849f1b2905825497b0dae P 79a93325cbb74798842ddee222b050ee [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79a93325cbb74798842ddee222b050ee, State: Initialized, Role: FOLLOWER
I20260929 10:57:51.515873 27175 consensus_queue.cc:260] T 6b61f157d65849f1b2905825497b0dae P 79a93325cbb74798842ddee222b050ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87a7fa2b136444c389cf89f4114fb9b4" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36489 } } peers { permanent_uuid: "79a93325cbb74798842ddee222b050ee" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 32973 } } peers { permanent_uuid: "4f941f11113449088d27274866fc18d2" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38929 } }
I20260929 10:57:51.516104 27175 ts_tablet_manager.cc:1428] T 6b61f157d65849f1b2905825497b0dae P 79a93325cbb74798842ddee222b050ee: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 10:57:51.516656 27095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.193@KRBTEST.COM'} at 127.10.238.193:52589
I20260929 10:57:51.517884 27283 heartbeater.cc:503] Master 127.10.238.254:39121 was elected leader, sending a full tablet report...
I20260929 10:57:51.521294 27291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:51.517312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36152 (local address 127.10.238.193:32973)
0929 10:57:51.518309 (+ 997us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:51.518313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:51.518324 (+ 11us) server_negotiation.cc:407] Connection header received
0929 10:57:51.518401 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:51.518403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:51.518417 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:51.518515 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:57:51.518676 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:51.519417 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:51.520357 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:51.520883 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:51.520973 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:51.521024 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:51.521176 (+ 152us) server_negotiation.cc:299] Negotiation successful
0929 10:57:51.521220 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":964,"thread_start_us":80,"threads_started":1}
I20260929 10:57:51.521293 27290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:51.517038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:32973 (local address 127.0.0.1:36152)
0929 10:57:51.517211 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:51.517247 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:51.517329 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:51.518520 (+ 1191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:57:51.518523 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:57:51.518525 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:57:51.518589 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:51.518592 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:51.519494 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:51.519494 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:51.520258 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:51.520260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:51.520971 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:51.520972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:51.521055 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:51.521070 (+ 15us) client_negotiation.cc:769] Sending connection context
0929 10:57:51.521148 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 10:57:51.521203 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":110,"threads_started":1}
I20260929 10:57:51.521922 27217 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b61f157d65849f1b2905825497b0dae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:57:31 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:36152
I20260929 10:57:51.522352 27292 tablet_replica.cc:330] stopping tablet replica
I20260929 10:57:51.522470 27292 raft_consensus.cc:2229] T 6b61f157d65849f1b2905825497b0dae P 79a93325cbb74798842ddee222b050ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:57:51.522533 27292 raft_consensus.cc:2258] T 6b61f157d65849f1b2905825497b0dae P 79a93325cbb74798842ddee222b050ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:57:51.522845 27292 ts_tablet_manager.cc:1905] T 6b61f157d65849f1b2905825497b0dae P 79a93325cbb74798842ddee222b050ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:57:51.524814 27292 ts_tablet_manager.cc:1918] T 6b61f157d65849f1b2905825497b0dae P 79a93325cbb74798842ddee222b050ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:57:51.524900 27292 log.cc:1196] T 6b61f157d65849f1b2905825497b0dae P 79a93325cbb74798842ddee222b050ee: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-0/wal/wals/6b61f157d65849f1b2905825497b0dae
I20260929 10:57:51.525116 27292 ts_tablet_manager.cc:1939] T 6b61f157d65849f1b2905825497b0dae P 79a93325cbb74798842ddee222b050ee: Deleting consensus metadata
I20260929 10:57:51.525630 27079 catalog_manager.cc:4841] TS 79a93325cbb74798842ddee222b050ee (127.10.238.193:32973): tablet 6b61f157d65849f1b2905825497b0dae (table test-table [id=2daa22959cdd47abaf2a0228d2f0d91e]) successfully deleted
W20260929 10:57:51.594192 27287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:57:51.594417 27287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:57:51.594442 27287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:57:51.595623 27287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:57:51.595666 27287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:57:51.595693 27287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.194
I20260929 10:57:51.597157 27287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:34957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab
--principal=kudu/127.10.238.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.194:38929
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.238.194
--webserver_port=38579
--tserver_master_addrs=127.10.238.254:39121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.194
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:57:51.597424 27287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:57:51.597661 27287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:57:51.602555 27300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:51.602571 27297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:51.602564 27298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:57:51.602855 27287 server_base.cc:1034] running on GCE node
I20260929 10:57:51.603062 27287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:57:51.603302 27287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:57:51.604470 27287 hybrid_clock.cc:648] HybridClock initialized: now 1790679471604457 us; error 23 us; skew 500 ppm
Sep 29 10:57:51 dist-test-slave-8nrt krb5kdc[26515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679471, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:57:51.606899 27287 init.cc:377] Logged in from keytab as kudu/127.10.238.194@KRBTEST.COM (short username kudu)
I20260929 10:57:51.607360 27287 webserver.cc:460] Webserver started at http://127.10.238.194:38579/ using document root <none> and password file <none>
I20260929 10:57:51.607582 27287 fs_manager.cc:359] Metadata directory not provided
I20260929 10:57:51.607662 27287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:57:51.608978 27287 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:57:51.609684 27307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:51.609856 27287 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:51.609953 27287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-1/wal
uuid: "4f941f11113449088d27274866fc18d2"
format_stamp: "Formatted at 2026-09-29 10:57:30 on dist-test-slave-8nrt"
server_key: "47e10221cc0709ba7dcda1a1822f25f3"
server_key_iv: "bb5440272c32eee47cacf91cb2b36556"
server_key_version: "encryptionkey@0"
I20260929 10:57:51.610299 27287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:57:51.621606 27287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:57:51.623525 27287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:57:51.623682 27287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:57:51.623917 27287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:57:51.624329 27314 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260929 10:57:51.625777 27314 ts_tablet_manager.cc:1852] T 6b61f157d65849f1b2905825497b0dae P 4f941f11113449088d27274866fc18d2: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260929 10:57:51.626039 27314 ts_tablet_manager.cc:1905] T 6b61f157d65849f1b2905825497b0dae P 4f941f11113449088d27274866fc18d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:57:51.628225 27314 ts_tablet_manager.cc:1918] T 6b61f157d65849f1b2905825497b0dae P 4f941f11113449088d27274866fc18d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:57:51.628296 27314 log.cc:1196] T 6b61f157d65849f1b2905825497b0dae P 4f941f11113449088d27274866fc18d2: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-1/wal/wals/6b61f157d65849f1b2905825497b0dae
I20260929 10:57:51.628587 27314 ts_tablet_manager.cc:1939] T 6b61f157d65849f1b2905825497b0dae P 4f941f11113449088d27274866fc18d2: Deleting consensus metadata
I20260929 10:57:51.628822 27287 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260929 10:57:51.628885 27287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260929 10:57:51.628947 27287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:57:51.628978 27287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:51.635083 27287 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.194:38929
I20260929 10:57:51.635138 27421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.194:38929 every 8 connection(s)
I20260929 10:57:51.635426 27287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
Sep 29 10:57:51 dist-test-slave-8nrt krb5kdc[26515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679471, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.194@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:57:51.642488 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 27287
I20260929 10:57:51.642757 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.195:36489
--local_ip_for_outbound_sockets=127.10.238.195
--tserver_master_addrs=127.10.238.254:39121
--webserver_port=36265
--webserver_interface=127.10.238.195
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab
--principal=kudu/127.10.238.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:34957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:57:51.644610 27286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:51.636107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:40651 (local address 127.10.238.254:39121)
0929 10:57:51.636165 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:51.636167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:51.636385 (+ 218us) server_negotiation.cc:407] Connection header received
0929 10:57:51.636908 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:51.636910 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:51.636926 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:51.636990 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:57:51.637691 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:51.638206 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:51.639222 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:51.639459 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:51.639539 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:51.641605 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:57:51.641606 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:57:51.641607 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:57:51.641613 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:57:51.643039 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:51.643533 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:51.643533 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:51.643534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:51.643584 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:51.643869 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:51.643870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:51.643870 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:51.643968 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:57:51.644017 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:51.644391 (+ 374us) server_negotiation.cc:299] Negotiation successful
0929 10:57:51.644491 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 10:57:51.644639 27424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:51.636102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.194:40651)
0929 10:57:51.636250 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:51.636286 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:51.636798 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:51.637061 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:57:51.637064 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:57:51.637391 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:57:51.637582 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:51.637587 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:51.638304 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:51.638305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:51.639127 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:51.639129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:51.639548 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:51.639549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:51.639671 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:51.640130 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:57:51.640138 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:57:51.641514 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:57:51.643112 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:51.643120 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:51.643122 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:51.643455 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:51.643644 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:51.643644 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:51.643645 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:51.643805 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:51.644028 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:57:51.644034 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:57:51.644225 (+ 191us) client_negotiation.cc:769] Sending connection context
0929 10:57:51.644351 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 10:57:51.644479 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":82,"threads_started":1}
I20260929 10:57:51.645454 27422 heartbeater.cc:347] Connected to a master server at 127.10.238.254:39121
I20260929 10:57:51.645562 27422 heartbeater.cc:464] Registering TS with master...
I20260929 10:57:51.645741 27422 heartbeater.cc:511] Master 127.10.238.254:39121 requested a full tablet report, sending...
I20260929 10:57:51.646162 27095 ts_manager.cc:194] Registered new tserver with Master: 4f941f11113449088d27274866fc18d2 (127.10.238.194:38929)
I20260929 10:57:51.646540 27095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.194@KRBTEST.COM'} at 127.10.238.194:40651
W20260929 10:57:51.724641 27425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:57:51.724835 27425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:57:51.724858 27425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:57:51.726056 27425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:57:51.726099 27425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:57:51.726125 27425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.195
I20260929 10:57:51.727664 27425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:34957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab
--principal=kudu/127.10.238.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.195:36489
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.238.195
--webserver_port=36265
--tserver_master_addrs=127.10.238.254:39121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.195
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:57:51.727881 27425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:57:51.728107 27425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:57:51.733306 27433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:51.733309 27431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:57:51.733301 27430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:57:51.733592 27425 server_base.cc:1034] running on GCE node
I20260929 10:57:51.733836 27425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:57:51.734131 27425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:57:51.735307 27425 hybrid_clock.cc:648] HybridClock initialized: now 1790679471735285 us; error 38 us; skew 500 ppm
Sep 29 10:57:51 dist-test-slave-8nrt krb5kdc[26515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679471, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:57:51.738190 27425 init.cc:377] Logged in from keytab as kudu/127.10.238.195@KRBTEST.COM (short username kudu)
I20260929 10:57:51.738590 27425 webserver.cc:460] Webserver started at http://127.10.238.195:36265/ using document root <none> and password file <none>
I20260929 10:57:51.738790 27425 fs_manager.cc:359] Metadata directory not provided
I20260929 10:57:51.738840 27425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:57:51.740231 27425 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:57:51.740831 27440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:51.741056 27425 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:57:51.741147 27425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-2/wal
uuid: "87a7fa2b136444c389cf89f4114fb9b4"
format_stamp: "Formatted at 2026-09-29 10:57:30 on dist-test-slave-8nrt"
server_key: "9607b613a2b05d9741da6f6055369ae1"
server_key_iv: "1b9fe9c5b638cf4aa669d4766b066079"
server_key_version: "encryptionkey@0"
I20260929 10:57:51.741542 27425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:57:51.750844 27425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:57:51.752694 27425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:57:51.752813 27425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:57:51.752990 27425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:57:51.753233 27425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:57:51.753320 27425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:51.753358 27425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:57:51.753371 27425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:57:51.759702 27425 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.195:36489
I20260929 10:57:51.759786 27553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.195:36489 every 8 connection(s)
I20260929 10:57:51.760051 27425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
Sep 29 10:57:51 dist-test-slave-8nrt krb5kdc[26515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679471, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.195@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:57:51.768554 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 27425
I20260929 10:57:51.769515 27286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:51.760784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:41383 (local address 127.10.238.254:39121)
0929 10:57:51.760831 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:51.760833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:51.761057 (+ 224us) server_negotiation.cc:407] Connection header received
0929 10:57:51.761631 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:51.761632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:51.761644 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:51.761698 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:57:51.762618 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:51.763160 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:51.764237 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:51.764418 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:51.764472 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:51.766610 (+ 2138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:57:51.766611 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:57:51.766611 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:57:51.766618 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:57:51.767895 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:51.768269 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:51.768270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:51.768270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:51.768318 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:51.768668 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:51.768669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:51.768670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:51.768836 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:57:51.768906 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:51.769372 (+ 466us) server_negotiation.cc:299] Negotiation successful
0929 10:57:51.769450 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 10:57:51.769681 27556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:51.760776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.195:41383)
0929 10:57:51.760934 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:51.760970 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:51.761541 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:51.761753 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:57:51.761756 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:57:51.762336 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:57:51.762524 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:51.762529 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:51.763268 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:51.763269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:51.764169 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:51.764172 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:51.764492 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:51.764492 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:51.764591 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:51.765072 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:57:51.765082 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:57:51.766528 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:57:51.767969 (+ 1441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:51.767975 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:51.767977 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:51.768206 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:51.768401 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:57:51.768403 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:57:51.768403 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:57:51.768556 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:57:51.768948 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:57:51.768955 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:57:51.769164 (+ 209us) client_negotiation.cc:769] Sending connection context
0929 10:57:51.769315 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 10:57:51.769479 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
I20260929 10:57:51.770228 27554 heartbeater.cc:347] Connected to a master server at 127.10.238.254:39121
I20260929 10:57:51.770344 27554 heartbeater.cc:464] Registering TS with master...
I20260929 10:57:51.770536 27554 heartbeater.cc:511] Master 127.10.238.254:39121 requested a full tablet report, sending...
I20260929 10:57:51.770908 27095 ts_manager.cc:194] Registered new tserver with Master: 87a7fa2b136444c389cf89f4114fb9b4 (127.10.238.195:36489)
I20260929 10:57:51.771273 27095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.195@KRBTEST.COM'} at 127.10.238.195:41383
I20260929 10:57:51.775414 27286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:51.769211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36274 (local address 127.10.238.254:39121)
0929 10:57:51.769612 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:51.769614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:51.769622 (+ 8us) server_negotiation.cc:407] Connection header received
0929 10:57:51.769644 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:51.769645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:51.769656 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:51.769711 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:57:51.770370 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:51.770906 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:51.771731 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:51.771942 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:51.772024 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:51.773092 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:57:51.773093 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:57:51.773094 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:57:51.773100 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:57:51.774312 (+ 1212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:51.774654 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:51.774655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:51.774656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:51.774707 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:51.774913 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:51.774914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:51.774915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:51.775024 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:57:51.775205 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:51.775240 (+ 35us) server_negotiation.cc:299] Negotiation successful
0929 10:57:51.775316 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":101,"threads_started":1}
I20260929 10:57:51.776094 11195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:57:51.778997 27286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:51.776433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36280 (local address 127.10.238.254:39121)
0929 10:57:51.776500 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:51.776502 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:51.776607 (+ 105us) server_negotiation.cc:407] Connection header received
0929 10:57:51.776720 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:51.776721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:51.776765 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:51.776846 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:57:51.776977 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:51.777524 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:51.778274 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:51.778470 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:51.778552 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:51.778718 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:57:51.778766 (+ 48us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0929 10:57:51.778935 (+ 169us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36280: token expired
Metrics: {"server-negotiator.queue_time_us":41}
W20260929 10:57:51.779076 27286 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36280: token expired
W20260929 10:57:51.779165 27024 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.10.238.254:39121)
I20260929 10:57:51.784576 27286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:51.779412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36286 (local address 127.10.238.254:39121)
0929 10:57:51.779464 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:51.779466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:51.779495 (+ 29us) server_negotiation.cc:407] Connection header received
0929 10:57:51.779540 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:51.779541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:51.779548 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:51.779592 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:57:51.780030 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:51.780451 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:51.781096 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:51.781349 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:51.781417 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:51.782428 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:57:51.782429 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:57:51.782430 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:57:51.782435 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:57:51.783591 (+ 1156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:51.783908 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:51.783909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:51.783909 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:51.783964 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:57:51.784133 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:57:51.784133 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:57:51.784133 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:57:51.784256 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:57:51.784391 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:51.784463 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 10:57:51.784523 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 10:57:51.785730 27095 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36286:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:57:51.787256 27095 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:57:51.789436 27217 tablet_service.cc:1467] Processing CreateTablet for tablet da6d9eabd9ec4a49927c636902746e7f (DEFAULT_TABLE table=test-table [id=e09f934537b044258290ae48c5e5df2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:57:51.789651 27217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da6d9eabd9ec4a49927c636902746e7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:57:51.791247 27175 tablet_bootstrap.cc:492] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee: Bootstrap starting.
I20260929 10:57:51.791607 27175 tablet_bootstrap.cc:654] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee: Neither blocks nor log segments found. Creating new log.
I20260929 10:57:51.792181 27175 tablet_bootstrap.cc:492] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee: No bootstrap required, opened a new log
I20260929 10:57:51.792238 27175 ts_tablet_manager.cc:1397] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260929 10:57:51.792362 27175 raft_consensus.cc:348] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f941f11113449088d27274866fc18d2" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38929 } } peers { permanent_uuid: "87a7fa2b136444c389cf89f4114fb9b4" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36489 } } peers { permanent_uuid: "79a93325cbb74798842ddee222b050ee" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 32973 } }
I20260929 10:57:51.792443 27175 raft_consensus.cc:374] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:57:51.792495 27175 raft_consensus.cc:729] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79a93325cbb74798842ddee222b050ee, State: Initialized, Role: FOLLOWER
I20260929 10:57:51.792539 27175 consensus_queue.cc:260] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f941f11113449088d27274866fc18d2" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38929 } } peers { permanent_uuid: "87a7fa2b136444c389cf89f4114fb9b4" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36489 } } peers { permanent_uuid: "79a93325cbb74798842ddee222b050ee" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 32973 } }
I20260929 10:57:51.792635 27175 ts_tablet_manager.cc:1428] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260929 10:57:51.793371 27290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:51.789075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.195:36489 (local address 127.0.0.1:58000)
0929 10:57:51.789168 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:51.789180 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:51.789215 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:51.789881 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:57:51.789882 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:57:51.789884 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:57:51.789918 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:51.789920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:51.791235 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:51.791236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:51.792117 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:51.792119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:51.793125 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:51.793125 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:51.793181 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:51.793184 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 10:57:51.793241 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 10:57:51.793309 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 10:57:51.793401 27562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:51.789419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58000 (local address 127.10.238.195:36489)
0929 10:57:51.789695 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:51.789699 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:51.789708 (+ 9us) server_negotiation.cc:407] Connection header received
0929 10:57:51.789764 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:51.789766 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:51.789783 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:51.789868 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:57:51.790061 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:51.791126 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:51.792330 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:51.793023 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:51.793125 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:51.793184 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:51.793283 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 10:57:51.793324 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":125,"threads_started":1}
I20260929 10:57:51.793496 27560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:51.789124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.194:38929 (local address 127.0.0.1:41732)
0929 10:57:51.789325 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:51.789333 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:51.789375 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:51.790214 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:57:51.790215 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:57:51.790218 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:57:51.790261 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:51.790263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:51.791093 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:51.791094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:51.792166 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:51.792169 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:51.793209 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:51.793210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:51.793294 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:51.793299 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 10:57:51.793329 (+ 30us) client_negotiation.cc:240] Negotiation successful
0929 10:57:51.793372 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":99,"threads_started":1}
I20260929 10:57:51.793705 27561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:51.789411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41732 (local address 127.10.238.194:38929)
0929 10:57:51.790040 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:51.790045 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:51.790054 (+ 9us) server_negotiation.cc:407] Connection header received
0929 10:57:51.790101 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:51.790103 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:51.790116 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:51.790215 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:57:51.790324 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:51.791002 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:51.792308 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:51.793116 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:51.793438 (+ 322us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:51.793503 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:51.793531 (+ 28us) server_negotiation.cc:299] Negotiation successful
0929 10:57:51.793579 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":109,"threads_started":1}
I20260929 10:57:51.794103 27488 tablet_service.cc:1467] Processing CreateTablet for tablet da6d9eabd9ec4a49927c636902746e7f (DEFAULT_TABLE table=test-table [id=e09f934537b044258290ae48c5e5df2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:57:51.794433 27488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da6d9eabd9ec4a49927c636902746e7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:57:51.797323 27564 tablet_bootstrap.cc:492] T da6d9eabd9ec4a49927c636902746e7f P 87a7fa2b136444c389cf89f4114fb9b4: Bootstrap starting.
I20260929 10:57:51.797861 27356 tablet_service.cc:1467] Processing CreateTablet for tablet da6d9eabd9ec4a49927c636902746e7f (DEFAULT_TABLE table=test-table [id=e09f934537b044258290ae48c5e5df2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:57:51.798159 27564 tablet_bootstrap.cc:654] T da6d9eabd9ec4a49927c636902746e7f P 87a7fa2b136444c389cf89f4114fb9b4: Neither blocks nor log segments found. Creating new log.
I20260929 10:57:51.798235 27356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da6d9eabd9ec4a49927c636902746e7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:57:51.798786 27564 log.cc:824] T da6d9eabd9ec4a49927c636902746e7f P 87a7fa2b136444c389cf89f4114fb9b4: Log is configured to *not* fsync() on all Append() calls
I20260929 10:57:51.799628 27564 tablet_bootstrap.cc:492] T da6d9eabd9ec4a49927c636902746e7f P 87a7fa2b136444c389cf89f4114fb9b4: No bootstrap required, opened a new log
I20260929 10:57:51.799711 27564 ts_tablet_manager.cc:1397] T da6d9eabd9ec4a49927c636902746e7f P 87a7fa2b136444c389cf89f4114fb9b4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 10:57:51.800594 27314 tablet_bootstrap.cc:492] T da6d9eabd9ec4a49927c636902746e7f P 4f941f11113449088d27274866fc18d2: Bootstrap starting.
I20260929 10:57:51.801419 27314 tablet_bootstrap.cc:654] T da6d9eabd9ec4a49927c636902746e7f P 4f941f11113449088d27274866fc18d2: Neither blocks nor log segments found. Creating new log.
I20260929 10:57:51.801733 27314 log.cc:824] T da6d9eabd9ec4a49927c636902746e7f P 4f941f11113449088d27274866fc18d2: Log is configured to *not* fsync() on all Append() calls
I20260929 10:57:51.801952 27564 raft_consensus.cc:348] T da6d9eabd9ec4a49927c636902746e7f P 87a7fa2b136444c389cf89f4114fb9b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f941f11113449088d27274866fc18d2" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38929 } } peers { permanent_uuid: "87a7fa2b136444c389cf89f4114fb9b4" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36489 } } peers { permanent_uuid: "79a93325cbb74798842ddee222b050ee" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 32973 } }
I20260929 10:57:51.802132 27564 raft_consensus.cc:374] T da6d9eabd9ec4a49927c636902746e7f P 87a7fa2b136444c389cf89f4114fb9b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:57:51.802183 27564 raft_consensus.cc:729] T da6d9eabd9ec4a49927c636902746e7f P 87a7fa2b136444c389cf89f4114fb9b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87a7fa2b136444c389cf89f4114fb9b4, State: Initialized, Role: FOLLOWER
I20260929 10:57:51.802299 27564 consensus_queue.cc:260] T da6d9eabd9ec4a49927c636902746e7f P 87a7fa2b136444c389cf89f4114fb9b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f941f11113449088d27274866fc18d2" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38929 } } peers { permanent_uuid: "87a7fa2b136444c389cf89f4114fb9b4" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36489 } } peers { permanent_uuid: "79a93325cbb74798842ddee222b050ee" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 32973 } }
I20260929 10:57:51.802438 27314 tablet_bootstrap.cc:492] T da6d9eabd9ec4a49927c636902746e7f P 4f941f11113449088d27274866fc18d2: No bootstrap required, opened a new log
I20260929 10:57:51.802497 27314 ts_tablet_manager.cc:1397] T da6d9eabd9ec4a49927c636902746e7f P 4f941f11113449088d27274866fc18d2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 10:57:51.802588 27564 ts_tablet_manager.cc:1428] T da6d9eabd9ec4a49927c636902746e7f P 87a7fa2b136444c389cf89f4114fb9b4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 10:57:51.802595 27554 heartbeater.cc:503] Master 127.10.238.254:39121 was elected leader, sending a full tablet report...
I20260929 10:57:51.804582 27314 raft_consensus.cc:348] T da6d9eabd9ec4a49927c636902746e7f P 4f941f11113449088d27274866fc18d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f941f11113449088d27274866fc18d2" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38929 } } peers { permanent_uuid: "87a7fa2b136444c389cf89f4114fb9b4" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36489 } } peers { permanent_uuid: "79a93325cbb74798842ddee222b050ee" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 32973 } }
I20260929 10:57:51.804701 27314 raft_consensus.cc:374] T da6d9eabd9ec4a49927c636902746e7f P 4f941f11113449088d27274866fc18d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:57:51.804723 27314 raft_consensus.cc:729] T da6d9eabd9ec4a49927c636902746e7f P 4f941f11113449088d27274866fc18d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f941f11113449088d27274866fc18d2, State: Initialized, Role: FOLLOWER
I20260929 10:57:51.804834 27314 consensus_queue.cc:260] T da6d9eabd9ec4a49927c636902746e7f P 4f941f11113449088d27274866fc18d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f941f11113449088d27274866fc18d2" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38929 } } peers { permanent_uuid: "87a7fa2b136444c389cf89f4114fb9b4" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36489 } } peers { permanent_uuid: "79a93325cbb74798842ddee222b050ee" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 32973 } }
I20260929 10:57:51.805054 27314 ts_tablet_manager.cc:1428] T da6d9eabd9ec4a49927c636902746e7f P 4f941f11113449088d27274866fc18d2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 10:57:51.805127 27422 heartbeater.cc:503] Master 127.10.238.254:39121 was elected leader, sending a full tablet report...
W20260929 10:57:51.885950 27423 tablet.cc:2369] T da6d9eabd9ec4a49927c636902746e7f P 4f941f11113449088d27274866fc18d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:57:51.901109 27567 raft_consensus.cc:482] T da6d9eabd9ec4a49927c636902746e7f P 87a7fa2b136444c389cf89f4114fb9b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:57:51.901238 27567 raft_consensus.cc:504] T da6d9eabd9ec4a49927c636902746e7f P 87a7fa2b136444c389cf89f4114fb9b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f941f11113449088d27274866fc18d2" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38929 } } peers { permanent_uuid: "87a7fa2b136444c389cf89f4114fb9b4" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36489 } } peers { permanent_uuid: "79a93325cbb74798842ddee222b050ee" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 32973 } }
I20260929 10:57:51.901646 27567 leader_election.cc:288] T da6d9eabd9ec4a49927c636902746e7f P 87a7fa2b136444c389cf89f4114fb9b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f941f11113449088d27274866fc18d2 (127.10.238.194:38929), 79a93325cbb74798842ddee222b050ee (127.10.238.193:32973)
I20260929 10:57:51.904698 27561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:51.901926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:49565 (local address 127.10.238.194:38929)
0929 10:57:51.902002 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:51.902004 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:51.902015 (+ 11us) server_negotiation.cc:407] Connection header received
0929 10:57:51.902045 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:51.902046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:51.902056 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:51.902123 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:57:51.902371 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:51.902885 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:51.903930 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:51.904305 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:51.904371 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:51.904378 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:51.904583 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 10:57:51.904630 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 10:57:51.904709 27556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:51.901822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.194:38929 (local address 127.10.238.195:49565)
0929 10:57:51.901884 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:51.901896 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:51.901942 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:51.902187 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:57:51.902188 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:57:51.902191 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:57:51.902282 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:51.902283 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:51.902969 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:51.902970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:51.903836 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:51.903838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:51.904398 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:51.904398 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:51.904499 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:51.904504 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 10:57:51.904565 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 10:57:51.904598 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":18}
I20260929 10:57:51.904784 27291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:51.901919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:51883 (local address 127.10.238.193:32973)
0929 10:57:51.902026 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:51.902028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:51.902162 (+ 134us) server_negotiation.cc:407] Connection header received
0929 10:57:51.902193 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:51.902194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:51.902205 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:51.902276 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:57:51.902463 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:51.902938 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:51.903964 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:51.904422 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:51.904493 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:51.904501 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:51.904672 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 10:57:51.904710 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 10:57:51.904862 27569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:51.901833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:32973 (local address 127.10.238.195:51883)
0929 10:57:51.902055 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:51.902064 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 10:57:51.902116 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:57:51.902285 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:57:51.902285 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:57:51.902287 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:57:51.902378 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:51.902380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:51.903018 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:51.903018 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:51.903842 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:57:51.903844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:57:51.904501 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:57:51.904502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:57:51.904583 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:51.904589 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 10:57:51.904646 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 10:57:51.904685 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":61,"thread_start_us":96,"threads_started":1}
I20260929 10:57:51.905148 27237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da6d9eabd9ec4a49927c636902746e7f" candidate_uuid: "87a7fa2b136444c389cf89f4114fb9b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79a93325cbb74798842ddee222b050ee" is_pre_election: true
I20260929 10:57:51.905180 27376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da6d9eabd9ec4a49927c636902746e7f" candidate_uuid: "87a7fa2b136444c389cf89f4114fb9b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f941f11113449088d27274866fc18d2" is_pre_election: true
I20260929 10:57:51.905342 27237 raft_consensus.cc:2454] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87a7fa2b136444c389cf89f4114fb9b4 in term 0.
I20260929 10:57:51.905344 27376 raft_consensus.cc:2454] T da6d9eabd9ec4a49927c636902746e7f P 4f941f11113449088d27274866fc18d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87a7fa2b136444c389cf89f4114fb9b4 in term 0.
I20260929 10:57:51.905604 27441 leader_election.cc:302] T da6d9eabd9ec4a49927c636902746e7f P 87a7fa2b136444c389cf89f4114fb9b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79a93325cbb74798842ddee222b050ee, 87a7fa2b136444c389cf89f4114fb9b4; no voters:
I20260929 10:57:51.905794 27567 raft_consensus.cc:2781] T da6d9eabd9ec4a49927c636902746e7f P 87a7fa2b136444c389cf89f4114fb9b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:57:51.905867 27567 raft_consensus.cc:482] T da6d9eabd9ec4a49927c636902746e7f P 87a7fa2b136444c389cf89f4114fb9b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:57:51.905903 27567 raft_consensus.cc:3037] T da6d9eabd9ec4a49927c636902746e7f P 87a7fa2b136444c389cf89f4114fb9b4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:57:51.906633 27567 raft_consensus.cc:504] T da6d9eabd9ec4a49927c636902746e7f P 87a7fa2b136444c389cf89f4114fb9b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f941f11113449088d27274866fc18d2" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38929 } } peers { permanent_uuid: "87a7fa2b136444c389cf89f4114fb9b4" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36489 } } peers { permanent_uuid: "79a93325cbb74798842ddee222b050ee" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 32973 } }
I20260929 10:57:51.906788 27567 leader_election.cc:288] T da6d9eabd9ec4a49927c636902746e7f P 87a7fa2b136444c389cf89f4114fb9b4 [CANDIDATE]: Term 1 election: Requested vote from peers 4f941f11113449088d27274866fc18d2 (127.10.238.194:38929), 79a93325cbb74798842ddee222b050ee (127.10.238.193:32973)
I20260929 10:57:51.906940 27376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da6d9eabd9ec4a49927c636902746e7f" candidate_uuid: "87a7fa2b136444c389cf89f4114fb9b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f941f11113449088d27274866fc18d2"
I20260929 10:57:51.906967 27237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da6d9eabd9ec4a49927c636902746e7f" candidate_uuid: "87a7fa2b136444c389cf89f4114fb9b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79a93325cbb74798842ddee222b050ee"
I20260929 10:57:51.907035 27237 raft_consensus.cc:3037] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:57:51.907032 27376 raft_consensus.cc:3037] T da6d9eabd9ec4a49927c636902746e7f P 4f941f11113449088d27274866fc18d2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:57:51.907995 27376 raft_consensus.cc:2454] T da6d9eabd9ec4a49927c636902746e7f P 4f941f11113449088d27274866fc18d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87a7fa2b136444c389cf89f4114fb9b4 in term 1.
I20260929 10:57:51.908219 27444 leader_election.cc:302] T da6d9eabd9ec4a49927c636902746e7f P 87a7fa2b136444c389cf89f4114fb9b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f941f11113449088d27274866fc18d2, 87a7fa2b136444c389cf89f4114fb9b4; no voters:
I20260929 10:57:51.908329 27567 raft_consensus.cc:2781] T da6d9eabd9ec4a49927c636902746e7f P 87a7fa2b136444c389cf89f4114fb9b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:57:51.908340 27237 raft_consensus.cc:2454] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87a7fa2b136444c389cf89f4114fb9b4 in term 1.
I20260929 10:57:51.908502 27567 raft_consensus.cc:686] T da6d9eabd9ec4a49927c636902746e7f P 87a7fa2b136444c389cf89f4114fb9b4 [term 1 LEADER]: Becoming Leader. State: Replica: 87a7fa2b136444c389cf89f4114fb9b4, State: Running, Role: LEADER
I20260929 10:57:51.908610 27567 consensus_queue.cc:237] T da6d9eabd9ec4a49927c636902746e7f P 87a7fa2b136444c389cf89f4114fb9b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f941f11113449088d27274866fc18d2" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38929 } } peers { permanent_uuid: "87a7fa2b136444c389cf89f4114fb9b4" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36489 } } peers { permanent_uuid: "79a93325cbb74798842ddee222b050ee" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 32973 } }
I20260929 10:57:51.909499 27095 catalog_manager.cc:5495] T da6d9eabd9ec4a49927c636902746e7f P 87a7fa2b136444c389cf89f4114fb9b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87a7fa2b136444c389cf89f4114fb9b4 (127.10.238.195). New cstate: current_term: 1 leader_uuid: "87a7fa2b136444c389cf89f4114fb9b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f941f11113449088d27274866fc18d2" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87a7fa2b136444c389cf89f4114fb9b4" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79a93325cbb74798842ddee222b050ee" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 32973 } health_report { overall_health: UNKNOWN } } }
I20260929 10:57:51.940241 27562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:51.938125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58010 (local address 127.10.238.195:36489)
0929 10:57:51.938190 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 10:57:51.938209 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 10:57:51.938218 (+ 9us) server_negotiation.cc:407] Connection header received
0929 10:57:51.938262 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:57:51.938263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:57:51.938280 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:57:51.938342 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:57:51.938462 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:51.938927 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:51.939604 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:57:51.939788 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:57:51.939837 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:57:51.939956 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:57:51.940051 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:57:51.940105 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 10:57:51.940141 (+ 36us) server_negotiation.cc:299] Negotiation successful
0929 10:57:51.940157 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 10:57:51.942111 27376 raft_consensus.cc:1261] T da6d9eabd9ec4a49927c636902746e7f P 4f941f11113449088d27274866fc18d2 [term 1 FOLLOWER]: Refusing update from remote peer 87a7fa2b136444c389cf89f4114fb9b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:57:51.942124 27237 raft_consensus.cc:1261] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [term 1 FOLLOWER]: Refusing update from remote peer 87a7fa2b136444c389cf89f4114fb9b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:57:51.942487 27567 consensus_queue.cc:1035] T da6d9eabd9ec4a49927c636902746e7f P 87a7fa2b136444c389cf89f4114fb9b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f941f11113449088d27274866fc18d2" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:57:51.942667 27570 consensus_queue.cc:1035] T da6d9eabd9ec4a49927c636902746e7f P 87a7fa2b136444c389cf89f4114fb9b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79a93325cbb74798842ddee222b050ee" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 32973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:57:51.945618 27574 mvcc.cc:205] Tried to move back new op lower bound from 7334623117072076800 to 7334623116938452992. Current Snapshot: MvccSnapshot[applied={T|T < 7334623117072076800}]
I20260929 10:57:51.947564 27095 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:57:51.947662 27095 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:57:51.948408 27095 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33: Sending DeleteTablet for 3 replicas of tablet da6d9eabd9ec4a49927c636902746e7f
I20260929 10:57:51.948711 27488 tablet_service.cc:1514] Processing DeleteTablet for tablet da6d9eabd9ec4a49927c636902746e7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:57:51 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:58000
I20260929 10:57:51.948781 27217 tablet_service.cc:1514] Processing DeleteTablet for tablet da6d9eabd9ec4a49927c636902746e7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:57:51 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:36152
I20260929 10:57:51.948794 27356 tablet_service.cc:1514] Processing DeleteTablet for tablet da6d9eabd9ec4a49927c636902746e7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:57:51 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:41732
I20260929 10:57:51.948868 27292 tablet_replica.cc:330] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee: stopping tablet replica
I20260929 10:57:51.948915 27292 raft_consensus.cc:2229] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:57:51.948962 27292 raft_consensus.cc:2258] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:57:51.949016 27584 tablet_replica.cc:330] T da6d9eabd9ec4a49927c636902746e7f P 4f941f11113449088d27274866fc18d2: stopping tablet replica
I20260929 10:57:51.949016 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 27133
I20260929 10:57:51.949115 27584 raft_consensus.cc:2229] T da6d9eabd9ec4a49927c636902746e7f P 4f941f11113449088d27274866fc18d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:57:51.949201 27584 raft_consensus.cc:2258] T da6d9eabd9ec4a49927c636902746e7f P 4f941f11113449088d27274866fc18d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:57:51.949738 27584 ts_tablet_manager.cc:1905] T da6d9eabd9ec4a49927c636902746e7f P 4f941f11113449088d27274866fc18d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:57:51.951551 27584 ts_tablet_manager.cc:1918] T da6d9eabd9ec4a49927c636902746e7f P 4f941f11113449088d27274866fc18d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:57:51.951613 27584 log.cc:1196] T da6d9eabd9ec4a49927c636902746e7f P 4f941f11113449088d27274866fc18d2: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-1/wal/wals/da6d9eabd9ec4a49927c636902746e7f
I20260929 10:57:51.951849 27584 ts_tablet_manager.cc:1939] T da6d9eabd9ec4a49927c636902746e7f P 4f941f11113449088d27274866fc18d2: Deleting consensus metadata
I20260929 10:57:51.952418 27082 catalog_manager.cc:4841] TS 4f941f11113449088d27274866fc18d2 (127.10.238.194:38929): tablet da6d9eabd9ec4a49927c636902746e7f (table test-table [id=e09f934537b044258290ae48c5e5df2b]) successfully deleted
W20260929 10:57:51.955088 27079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.238.193:32973 (error 108)
I20260929 10:57:51.955487 27583 tablet_replica.cc:330] T da6d9eabd9ec4a49927c636902746e7f P 87a7fa2b136444c389cf89f4114fb9b4: stopping tablet replica
I20260929 10:57:51.955595 27583 raft_consensus.cc:2229] T da6d9eabd9ec4a49927c636902746e7f P 87a7fa2b136444c389cf89f4114fb9b4 [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:57:51.955619 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 27287
I20260929 10:57:51.955662 27560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:57:51.955533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:32973 (local address 127.0.0.1:36156)
0929 10:57:51.955586 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0929 10:57:51.955629 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.193:32973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260929 10:57:51.955704 27583 raft_consensus.cc:2258] T da6d9eabd9ec4a49927c636902746e7f P 87a7fa2b136444c389cf89f4114fb9b4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 10:57:51.956424 27079 catalog_manager.cc:4584] TS 79a93325cbb74798842ddee222b050ee (127.10.238.193:32973): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet da6d9eabd9ec4a49927c636902746e7f: Network error: Client connection negotiation failed: client connection to 127.10.238.193:32973: connect: Connection refused (error 111)
I20260929 10:57:51.957414 27583 ts_tablet_manager.cc:1905] T da6d9eabd9ec4a49927c636902746e7f P 87a7fa2b136444c389cf89f4114fb9b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:57:51.959215 27583 ts_tablet_manager.cc:1918] T da6d9eabd9ec4a49927c636902746e7f P 87a7fa2b136444c389cf89f4114fb9b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:57:51.959293 27583 log.cc:1196] T da6d9eabd9ec4a49927c636902746e7f P 87a7fa2b136444c389cf89f4114fb9b4: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-2/wal/wals/da6d9eabd9ec4a49927c636902746e7f
I20260929 10:57:51.959578 27583 ts_tablet_manager.cc:1939] T da6d9eabd9ec4a49927c636902746e7f P 87a7fa2b136444c389cf89f4114fb9b4: Deleting consensus metadata
I20260929 10:57:51.959975 27080 catalog_manager.cc:4841] TS 87a7fa2b136444c389cf89f4114fb9b4 (127.10.238.195:36489): tablet da6d9eabd9ec4a49927c636902746e7f (table test-table [id=e09f934537b044258290ae48c5e5df2b]) successfully deleted
I20260929 10:57:51.961022 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 27425
I20260929 10:57:51.966701 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 27062
I20260929 10:58:11.972841 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:39121
--webserver_interface=127.10.238.254
--webserver_port=44857
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:34957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:39121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:58:12.067247 27586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:58:12.067515 27586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:58:12.067559 27586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:58:12.068718 27586 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 10:58:12.068787 27586 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 10:58:12.068809 27586 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 10:58:12.068827 27586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:58:12.068861 27586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:58:12.068885 27586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:58:12.068917 27586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:58:12.068948 27586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:58:12.070580 27586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:34957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:39121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:39121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.238.254
--webserver_port=44857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:58:12.070924 27586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:58:12.071161 27586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 10:58:12.077052 27586 server_base.cc:1034] running on GCE node
W20260929 10:58:12.076996 27594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:12.077013 27592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:12.076996 27591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:12.077513 27586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:58:12.077888 27586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:58:12.079080 27586 hybrid_clock.cc:648] HybridClock initialized: now 1790679492079061 us; error 38 us; skew 500 ppm
Sep 29 10:58:12 dist-test-slave-8nrt krb5kdc[26515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679492, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:58:12.085479 27586 init.cc:377] Logged in from keytab as kudu/127.10.238.254@KRBTEST.COM (short username kudu)
I20260929 10:58:12.085875 27586 webserver.cc:460] Webserver started at http://127.10.238.254:44857/ using document root <none> and password file <none>
I20260929 10:58:12.086140 27586 fs_manager.cc:359] Metadata directory not provided
I20260929 10:58:12.086186 27586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:58:12.087594 27586 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:58:12.088383 27601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:12.088686 27586 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:58:12.088785 27586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "1a72d564424948f3ac82f26f9e88be33"
format_stamp: "Formatted at 2026-09-29 10:57:30 on dist-test-slave-8nrt"
server_key: "ab9ad5a75f9560d4645d3da831e19da7"
server_key_iv: "1eca75655fecc6209f71c1ed9d455d96"
server_key_version: "encryptionkey@0"
I20260929 10:58:12.089195 27586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:58:12.109331 27586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:58:12.111387 27586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:58:12.111550 27586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:58:12.116477 27586 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.254:39121
I20260929 10:58:12.116547 27653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.254:39121 every 8 connection(s)
I20260929 10:58:12.116961 27586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:58:12.118642 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 27586
I20260929 10:58:12.118988 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.193:32973
--local_ip_for_outbound_sockets=127.10.238.193
--tserver_master_addrs=127.10.238.254:39121
--webserver_port=32913
--webserver_interface=127.10.238.193
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab
--principal=kudu/127.10.238.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:34957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:58:12.119109 27654 sys_catalog.cc:263] Verifying existing consensus state
I20260929 10:58:12.120234 27654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33: Bootstrap starting.
I20260929 10:58:12.131572 27654 log.cc:824] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33: Log is configured to *not* fsync() on all Append() calls
I20260929 10:58:12.134229 27654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 10:58:12.134548 27654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33: Bootstrap complete.
I20260929 10:58:12.136981 27654 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a72d564424948f3ac82f26f9e88be33" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39121 } }
I20260929 10:58:12.137233 27654 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a72d564424948f3ac82f26f9e88be33, State: Initialized, Role: FOLLOWER
I20260929 10:58:12.137444 27654 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a72d564424948f3ac82f26f9e88be33" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39121 } }
I20260929 10:58:12.137529 27654 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:58:12.137583 27654 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:58:12.137662 27654 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33 [term 2 FOLLOWER]: Advancing to term 3
I20260929 10:58:12.138701 27654 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a72d564424948f3ac82f26f9e88be33" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39121 } }
I20260929 10:58:12.138818 27654 leader_election.cc:302] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a72d564424948f3ac82f26f9e88be33; no voters:
I20260929 10:58:12.139031 27654 leader_election.cc:288] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 10:58:12.139066 27658 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 10:58:12.139192 27658 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33 [term 3 LEADER]: Becoming Leader. State: Replica: 1a72d564424948f3ac82f26f9e88be33, State: Running, Role: LEADER
I20260929 10:58:12.139304 27658 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a72d564424948f3ac82f26f9e88be33" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39121 } }
I20260929 10:58:12.139415 27654 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:58:12.139631 27658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a72d564424948f3ac82f26f9e88be33. Latest consensus state: current_term: 3 leader_uuid: "1a72d564424948f3ac82f26f9e88be33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a72d564424948f3ac82f26f9e88be33" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39121 } } }
I20260929 10:58:12.139720 27658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33 [sys.catalog]: This master's current role is: LEADER
I20260929 10:58:12.139631 27659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "1a72d564424948f3ac82f26f9e88be33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a72d564424948f3ac82f26f9e88be33" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39121 } } }
I20260929 10:58:12.139894 27659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33 [sys.catalog]: This master's current role is: LEADER
I20260929 10:58:12.139910 27662 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:58:12.141075 27662 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:58:12.141216 27662 catalog_manager.cc:1259] Loaded cluster ID: cc670bf9d7ce42b9867539bb7bf6fe0b
I20260929 10:58:12.141543 27662 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:58:12.142407 27662 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:58:12.142635 27662 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33: Loaded TSK: 0
I20260929 10:58:12.142699 27662 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1a72d564424948f3ac82f26f9e88be33: Loaded TSK: 1
F20260929 10:58:12.142956 27662 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f45412a89ad google::LogMessage::Fail() at ??:0
@ 0x7f45412ab88a google::LogMessage::SendToLog() at ??:0
@ 0x7f45412a84de google::LogMessage::Flush() at ??:0
@ 0x7f45412a9a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f4543a15104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f4543a12d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f45492021cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f454920248e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f4549202594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f45491c8c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f45491d2ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f45431760d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f454316b954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f4543fcf6db start_thread at ??:0
@ 0x7f453fda661f clone at ??:0
W20260929 10:58:12.244169 27656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:58:12.244445 27656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:58:12.244483 27656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:58:12.246237 27656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:58:12.246299 27656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:58:12.246338 27656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.193
I20260929 10:58:12.248405 27656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:34957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab
--principal=kudu/127.10.238.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.193:32973
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.238.193
--webserver_port=32913
--tserver_master_addrs=127.10.238.254:39121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.193
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:58:12.248649 27656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:58:12.248934 27656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:58:12.254544 27695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:12.254590 27696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:12.254746 27656 server_base.cc:1034] running on GCE node
W20260929 10:58:12.254874 27698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:12.255085 27656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:58:12.255352 27656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:58:12.256518 27656 hybrid_clock.cc:648] HybridClock initialized: now 1790679492256504 us; error 32 us; skew 500 ppm
Sep 29 10:58:12 dist-test-slave-8nrt krb5kdc[26515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679492, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:58:12.259215 27656 init.cc:377] Logged in from keytab as kudu/127.10.238.193@KRBTEST.COM (short username kudu)
I20260929 10:58:12.259727 27656 webserver.cc:460] Webserver started at http://127.10.238.193:32913/ using document root <none> and password file <none>
I20260929 10:58:12.260123 27656 fs_manager.cc:359] Metadata directory not provided
I20260929 10:58:12.260198 27656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:58:12.261571 27656 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:58:12.263626 27705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:12.263813 27656 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 10:58:12.263911 27656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-0/wal
uuid: "79a93325cbb74798842ddee222b050ee"
format_stamp: "Formatted at 2026-09-29 10:57:30 on dist-test-slave-8nrt"
server_key: "19c884ac07aaad4c5b3d4aeefec5a056"
server_key_iv: "76bccd5b2f16523b5cb68ed493613b48"
server_key_version: "encryptionkey@0"
I20260929 10:58:12.264278 27656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:58:12.286883 27656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:58:12.289225 27656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:58:12.289448 27656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:58:12.289697 27656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:58:12.290227 27712 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 10:58:12.292186 27656 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 10:58:12.292264 27656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260929 10:58:12.292307 27656 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 10:58:12.292961 27656 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 10:58:12.293020 27656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260929 10:58:12.293151 27712 tablet_bootstrap.cc:492] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee: Bootstrap starting.
I20260929 10:58:12.301573 27656 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.193:32973
I20260929 10:58:12.301976 27656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
I20260929 10:58:12.302399 27819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.193:32973 every 8 connection(s)
I20260929 10:58:12.308604 27712 log.cc:824] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee: Log is configured to *not* fsync() on all Append() calls
I20260929 10:58:12.308983 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 27656
I20260929 10:58:12.309356 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.194:38929
--local_ip_for_outbound_sockets=127.10.238.194
--tserver_master_addrs=127.10.238.254:39121
--webserver_port=38579
--webserver_interface=127.10.238.194
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab
--principal=kudu/127.10.238.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:34957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:58:12.311508 27712 tablet_bootstrap.cc:492] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 10:58:12.311813 27712 tablet_bootstrap.cc:492] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee: Bootstrap complete.
I20260929 10:58:12.312804 27712 ts_tablet_manager.cc:1397] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.009s
I20260929 10:58:12.314079 27712 raft_consensus.cc:348] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f941f11113449088d27274866fc18d2" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38929 } } peers { permanent_uuid: "87a7fa2b136444c389cf89f4114fb9b4" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36489 } } peers { permanent_uuid: "79a93325cbb74798842ddee222b050ee" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 32973 } }
I20260929 10:58:12.314345 27712 raft_consensus.cc:729] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79a93325cbb74798842ddee222b050ee, State: Initialized, Role: FOLLOWER
I20260929 10:58:12.314529 27712 consensus_queue.cc:260] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f941f11113449088d27274866fc18d2" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38929 } } peers { permanent_uuid: "87a7fa2b136444c389cf89f4114fb9b4" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36489 } } peers { permanent_uuid: "79a93325cbb74798842ddee222b050ee" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 32973 } }
I20260929 10:58:12.314896 27712 ts_tablet_manager.cc:1428] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
W20260929 10:58:12.386776 27822 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:58:12.302914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.193:44479)
0929 10:58:12.303104 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:12.303142 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 10:58:12.303616 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:58:12.386547 (+ 82931us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39121: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":101,"threads_started":1}
W20260929 10:58:12.387055 27820 heartbeater.cc:644] Failed to heartbeat to 127.10.238.254:39121 (0 consecutive failures): Network error: Failed to ping master at 127.10.238.254:39121: Client connection negotiation failed: client connection to 127.10.238.254:39121: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 10:58:12.387409 27822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:12.387276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.193:53961)
0929 10:58:12.387335 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:12.387377 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 10:58:12.387813 27822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:12.387726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.193:34027)
0929 10:58:12.387774 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:12.387798 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 10:58:12.387951 27820 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 10:58:12.410130 27824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:58:12.410346 27824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:58:12.410370 27824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:58:12.411482 27824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:58:12.411515 27824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:58:12.411540 27824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.194
I20260929 10:58:12.412930 27824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:34957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab
--principal=kudu/127.10.238.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.194:38929
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.238.194
--webserver_port=38579
--tserver_master_addrs=127.10.238.254:39121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.194
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:58:12.413116 27824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:58:12.413353 27824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:58:12.418516 27833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:12.418623 27824 server_base.cc:1034] running on GCE node
W20260929 10:58:12.418533 27830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:12.418601 27831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:12.419003 27824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:58:12.419250 27824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:58:12.420388 27824 hybrid_clock.cc:648] HybridClock initialized: now 1790679492420377 us; error 25 us; skew 500 ppm
Sep 29 10:58:12 dist-test-slave-8nrt krb5kdc[26515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679492, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:58:12.422920 27824 init.cc:377] Logged in from keytab as kudu/127.10.238.194@KRBTEST.COM (short username kudu)
I20260929 10:58:12.423300 27824 webserver.cc:460] Webserver started at http://127.10.238.194:38579/ using document root <none> and password file <none>
I20260929 10:58:12.423489 27824 fs_manager.cc:359] Metadata directory not provided
I20260929 10:58:12.423532 27824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:58:12.424795 27824 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:58:12.425418 27840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:12.425593 27824 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 10:58:12.425665 27824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-1/wal
uuid: "4f941f11113449088d27274866fc18d2"
format_stamp: "Formatted at 2026-09-29 10:57:30 on dist-test-slave-8nrt"
server_key: "47e10221cc0709ba7dcda1a1822f25f3"
server_key_iv: "bb5440272c32eee47cacf91cb2b36556"
server_key_version: "encryptionkey@0"
I20260929 10:58:12.426008 27824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:58:12.460786 27824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:58:12.462888 27824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:58:12.463006 27824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:58:12.463194 27824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:58:12.463446 27824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:58:12.463493 27824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:12.463533 27824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:58:12.463546 27824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:12.469606 27824 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.194:38929
I20260929 10:58:12.469712 27953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.194:38929 every 8 connection(s)
I20260929 10:58:12.469954 27824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
I20260929 10:58:12.471019 27956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:12.470639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.194:52163)
0929 10:58:12.470799 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:12.470873 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":75,"threads_started":1}
W20260929 10:58:12.471472 27954 heartbeater.cc:644] Failed to heartbeat to 127.10.238.254:39121 (0 consecutive failures): Network error: Failed to ping master at 127.10.238.254:39121: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
I20260929 10:58:12.471772 27956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:12.471681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.194:33087)
0929 10:58:12.471729 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:12.471755 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 10:58:12.472064 27956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:12.471988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.194:52943)
0929 10:58:12.472028 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:12.472049 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260929 10:58:12.472182 27954 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 10:58:12.476241 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 27824
I20260929 10:58:12.476583 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.195:36489
--local_ip_for_outbound_sockets=127.10.238.195
--tserver_master_addrs=127.10.238.254:39121
--webserver_port=36265
--webserver_interface=127.10.238.195
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab
--principal=kudu/127.10.238.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:34957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:58:12.562338 27957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:58:12.562561 27957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:58:12.562599 27957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:58:12.563776 27957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:58:12.563844 27957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:58:12.563890 27957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.195
I20260929 10:58:12.565385 27957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:34957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab
--principal=kudu/127.10.238.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.195:36489
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.238.195
--webserver_port=36265
--tserver_master_addrs=127.10.238.254:39121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.195
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:58:12.565660 27957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:58:12.565903 27957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:58:12.571017 27965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:12.571108 27957 server_base.cc:1034] running on GCE node
W20260929 10:58:12.571034 27962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:12.571034 27963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:12.571445 27957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:58:12.571703 27957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:58:12.572880 27957 hybrid_clock.cc:648] HybridClock initialized: now 1790679492572861 us; error 31 us; skew 500 ppm
Sep 29 10:58:12 dist-test-slave-8nrt krb5kdc[26515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679492, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:58:12.575430 27957 init.cc:377] Logged in from keytab as kudu/127.10.238.195@KRBTEST.COM (short username kudu)
I20260929 10:58:12.575919 27957 webserver.cc:460] Webserver started at http://127.10.238.195:36265/ using document root <none> and password file <none>
I20260929 10:58:12.576156 27957 fs_manager.cc:359] Metadata directory not provided
I20260929 10:58:12.576226 27957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:58:12.577685 27957 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260929 10:58:12.578263 27972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:12.578444 27957 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:12.578547 27957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-2/wal
uuid: "87a7fa2b136444c389cf89f4114fb9b4"
format_stamp: "Formatted at 2026-09-29 10:57:30 on dist-test-slave-8nrt"
server_key: "9607b613a2b05d9741da6f6055369ae1"
server_key_iv: "1b9fe9c5b638cf4aa669d4766b066079"
server_key_version: "encryptionkey@0"
I20260929 10:58:12.578895 27957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:58:12.586282 27957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:58:12.588625 27957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:58:12.588784 27957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:58:12.588994 27957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:58:12.589334 27957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:58:12.589396 27957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:12.589437 27957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:58:12.589462 27957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:12.595366 27957 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.195:36489
I20260929 10:58:12.595410 28085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.195:36489 every 8 connection(s)
I20260929 10:58:12.595731 27957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
I20260929 10:58:12.596748 28088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:12.596379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.195:49109)
0929 10:58:12.596542 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:12.596610 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":71,"threads_started":1}
W20260929 10:58:12.597168 28086 heartbeater.cc:644] Failed to heartbeat to 127.10.238.254:39121 (0 consecutive failures): Network error: Failed to ping master at 127.10.238.254:39121: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
I20260929 10:58:12.597464 28088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:12.597376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.195:36597)
0929 10:58:12.597422 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:12.597448 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 10:58:12.597759 28088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:12.597681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.195:47861)
0929 10:58:12.597720 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:12.597744 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260929 10:58:12.597882 28086 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 10:58:12.601784 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 27957
I20260929 10:58:13.388577 28090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:13.388331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.193:42473)
0929 10:58:13.388472 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:13.388513 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":61,"threads_started":1}
I20260929 10:58:13.459954 28091 raft_consensus.cc:482] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:58:13.460071 28091 raft_consensus.cc:504] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f941f11113449088d27274866fc18d2" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38929 } } peers { permanent_uuid: "87a7fa2b136444c389cf89f4114fb9b4" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36489 } } peers { permanent_uuid: "79a93325cbb74798842ddee222b050ee" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 32973 } }
I20260929 10:58:13.460391 28091 leader_election.cc:288] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4f941f11113449088d27274866fc18d2 (127.10.238.194:38929), 87a7fa2b136444c389cf89f4114fb9b4 (127.10.238.195:36489)
Sep 29 10:58:13 dist-test-slave-8nrt krb5kdc[26515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679492, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.193@KRBTEST.COM for kudu/127.10.238.194@KRBTEST.COM
Sep 29 10:58:13 dist-test-slave-8nrt krb5kdc[26515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679492, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.193@KRBTEST.COM for kudu/127.10.238.195@KRBTEST.COM
I20260929 10:58:13.470950 28093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:13.460657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:43999 (local address 127.10.238.194:38929)
0929 10:58:13.460853 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:13.460866 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:13.460893 (+ 27us) server_negotiation.cc:407] Connection header received
0929 10:58:13.461398 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:13.461401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:13.461456 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:13.461607 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:13.462321 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:13.463482 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:13.464551 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:13.464976 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:13.465099 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:13.467373 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:13.467377 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:13.467377 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:58:13.467389 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:13.469026 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:13.469573 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:13.469576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:13.469577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:13.469677 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:13.469986 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:13.469987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:13.469987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:13.470262 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:13.470357 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:13.470698 (+ 341us) server_negotiation.cc:299] Negotiation successful
0929 10:58:13.470846 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":62,"threads_started":1}
I20260929 10:58:13.470961 28090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:13.460520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.194:38929 (local address 127.10.238.193:43999)
0929 10:58:13.460608 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:13.460622 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:58:13.460854 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:58:13.461637 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:58:13.461640 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:58:13.462016 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:58:13.462202 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:13.462208 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:13.463606 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:13.463607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:13.464479 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:13.464481 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:13.465066 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:13.465067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:13.465216 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:13.465799 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:58:13.465814 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:58:13.467254 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:58:13.469125 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:58:13.469132 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:58:13.469133 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:58:13.469483 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:58:13.469762 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:58:13.469763 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:58:13.469763 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:58:13.469909 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:58:13.470350 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:58:13.470357 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:58:13.470504 (+ 147us) client_negotiation.cc:769] Sending connection context
0929 10:58:13.470627 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 10:58:13.470795 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":2}
W20260929 10:58:13.471799 27709 leader_election.cc:341] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4f941f11113449088d27274866fc18d2 (127.10.238.194:38929): Not found: Tablet not found: da6d9eabd9ec4a49927c636902746e7f
I20260929 10:58:13.472271 28092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:13.460554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.195:36489 (local address 127.10.238.193:45015)
0929 10:58:13.460756 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:13.460765 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 10:58:13.460811 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:58:13.461637 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:58:13.461640 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:58:13.462018 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:58:13.462203 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:13.462208 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:13.465616 (+ 3408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:13.465617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:13.466089 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:13.466091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:13.466731 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:13.466732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:13.466787 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:13.467243 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:58:13.467248 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:58:13.468692 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:58:13.470908 (+ 2216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:58:13.470909 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:58:13.470909 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:58:13.471123 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:58:13.471446 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:58:13.471447 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:58:13.471447 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:58:13.471514 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:58:13.471937 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:58:13.471938 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:58:13.471982 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 10:58:13.472070 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 10:58:13.472166 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":5,"thread_start_us":53,"threads_started":1}
I20260929 10:58:13.472461 28094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:13.460735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:45015 (local address 127.10.238.195:36489)
0929 10:58:13.460907 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:13.460915 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:13.460932 (+ 17us) server_negotiation.cc:407] Connection header received
0929 10:58:13.461418 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:13.461421 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:13.461471 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:13.461610 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:13.462321 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:13.465531 (+ 3210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:13.466187 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:13.466612 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:13.466761 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:13.468801 (+ 2040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:13.468805 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:13.468805 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:58:13.468815 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:13.470797 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:13.471245 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:13.471248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:13.471248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:13.471365 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:13.471587 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:13.471587 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:13.471588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:13.471857 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:13.471940 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:13.472132 (+ 192us) server_negotiation.cc:299] Negotiation successful
0929 10:58:13.472257 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":55,"threads_started":1}
I20260929 10:58:13.472784 28095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:13.472567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.194:46675)
0929 10:58:13.472678 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:13.472742 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":64,"threads_started":1}
W20260929 10:58:13.473050 27707 leader_election.cc:341] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87a7fa2b136444c389cf89f4114fb9b4 (127.10.238.195:36489): Not found: Tablet not found: da6d9eabd9ec4a49927c636902746e7f
I20260929 10:58:13.473114 27707 leader_election.cc:302] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 79a93325cbb74798842ddee222b050ee; no voters: 4f941f11113449088d27274866fc18d2, 87a7fa2b136444c389cf89f4114fb9b4
I20260929 10:58:13.473249 28091 raft_consensus.cc:2726] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 10:58:13.598402 28096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:13.598196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.195:56521)
0929 10:58:13.598318 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:13.598367 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":57,"threads_started":1}
I20260929 10:58:14.389588 28097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:14.389339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.193:41877)
0929 10:58:14.389496 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:14.389542 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":76,"threads_started":1}
I20260929 10:58:14.473501 28098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:14.473186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.194:55223)
0929 10:58:14.473365 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:14.473424 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":101,"threads_started":1}
I20260929 10:58:14.599077 28099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:14.598847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.195:58855)
0929 10:58:14.598992 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:14.599041 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":58,"threads_started":1}
I20260929 10:58:15.113530 28100 raft_consensus.cc:482] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:58:15.113624 28100 raft_consensus.cc:504] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f941f11113449088d27274866fc18d2" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38929 } } peers { permanent_uuid: "87a7fa2b136444c389cf89f4114fb9b4" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36489 } } peers { permanent_uuid: "79a93325cbb74798842ddee222b050ee" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 32973 } }
I20260929 10:58:15.113842 28100 leader_election.cc:288] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4f941f11113449088d27274866fc18d2 (127.10.238.194:38929), 87a7fa2b136444c389cf89f4114fb9b4 (127.10.238.195:36489)
W20260929 10:58:15.114296 27707 leader_election.cc:341] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87a7fa2b136444c389cf89f4114fb9b4 (127.10.238.195:36489): Not found: Tablet not found: da6d9eabd9ec4a49927c636902746e7f
W20260929 10:58:15.114396 27709 leader_election.cc:341] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4f941f11113449088d27274866fc18d2 (127.10.238.194:38929): Not found: Tablet not found: da6d9eabd9ec4a49927c636902746e7f
I20260929 10:58:15.114488 27709 leader_election.cc:302] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 79a93325cbb74798842ddee222b050ee; no voters: 4f941f11113449088d27274866fc18d2, 87a7fa2b136444c389cf89f4114fb9b4
I20260929 10:58:15.114555 28100 raft_consensus.cc:2726] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 10:58:15.390264 28101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:15.390007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.193:59231)
0929 10:58:15.390126 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:15.390196 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":57,"threads_started":1}
I20260929 10:58:15.474133 28102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:15.473927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.194:45259)
0929 10:58:15.474041 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:15.474079 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":58,"threads_started":1}
I20260929 10:58:15.599761 28103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:15.599559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.195:51413)
0929 10:58:15.599677 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:15.599723 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":62,"threads_started":1}
I20260929 10:58:16.391000 28104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:16.390766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.193:47999)
0929 10:58:16.390915 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:16.390954 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":62,"threads_started":1}
I20260929 10:58:16.474812 28105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:16.474589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.194:56755)
0929 10:58:16.474722 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:16.474764 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":59,"threads_started":1}
I20260929 10:58:16.600437 28106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:16.600221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.195:57725)
0929 10:58:16.600353 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:16.600393 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":59,"threads_started":1}
I20260929 10:58:17.391772 28107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:17.391547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.193:49873)
0929 10:58:17.391687 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:17.391726 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":93,"threads_started":1}
I20260929 10:58:17.432735 28108 raft_consensus.cc:482] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:58:17.432828 28108 raft_consensus.cc:504] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f941f11113449088d27274866fc18d2" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38929 } } peers { permanent_uuid: "87a7fa2b136444c389cf89f4114fb9b4" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36489 } } peers { permanent_uuid: "79a93325cbb74798842ddee222b050ee" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 32973 } }
I20260929 10:58:17.432989 28108 leader_election.cc:288] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4f941f11113449088d27274866fc18d2 (127.10.238.194:38929), 87a7fa2b136444c389cf89f4114fb9b4 (127.10.238.195:36489)
W20260929 10:58:17.433419 27707 leader_election.cc:341] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87a7fa2b136444c389cf89f4114fb9b4 (127.10.238.195:36489): Not found: Tablet not found: da6d9eabd9ec4a49927c636902746e7f
W20260929 10:58:17.433521 27709 leader_election.cc:341] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4f941f11113449088d27274866fc18d2 (127.10.238.194:38929): Not found: Tablet not found: da6d9eabd9ec4a49927c636902746e7f
I20260929 10:58:17.433607 27709 leader_election.cc:302] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 79a93325cbb74798842ddee222b050ee; no voters: 4f941f11113449088d27274866fc18d2, 87a7fa2b136444c389cf89f4114fb9b4
I20260929 10:58:17.433725 28108 raft_consensus.cc:2726] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 10:58:17.475493 28109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:17.475281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.194:34963)
0929 10:58:17.475414 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:17.475452 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":56,"threads_started":1}
I20260929 10:58:17.601153 28110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:17.600954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.195:47463)
0929 10:58:17.601083 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:17.601115 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":70,"threads_started":1}
I20260929 10:58:18.392659 28111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:18.392375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.193:35723)
0929 10:58:18.392533 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:18.392576 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":65,"threads_started":1}
I20260929 10:58:18.476162 28112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:18.475927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.194:50249)
0929 10:58:18.476082 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:18.476120 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":72,"threads_started":1}
I20260929 10:58:18.601818 28113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:18.601621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.195:60613)
0929 10:58:18.601743 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:18.601778 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":57,"threads_started":1}
I20260929 10:58:19.393474 28114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:19.393204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.193:36205)
0929 10:58:19.393388 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:19.393427 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":135,"threads_started":1}
I20260929 10:58:19.476895 28115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:19.476668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.194:48965)
0929 10:58:19.476809 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:19.476850 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":81,"threads_started":1}
I20260929 10:58:19.602497 28116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:19.602277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.195:45053)
0929 10:58:19.602412 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:19.602450 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":59,"threads_started":1}
I20260929 10:58:20.394181 28117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:20.393958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.193:44943)
0929 10:58:20.394090 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:20.394132 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":66,"threads_started":1}
I20260929 10:58:20.477588 28118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:20.477382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.194:54927)
0929 10:58:20.477502 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:20.477538 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":62,"threads_started":1}
I20260929 10:58:20.603154 28119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:20.602942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.195:57211)
0929 10:58:20.603077 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:20.603113 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":58,"threads_started":1}
I20260929 10:58:21.394889 28120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:21.394682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.193:60865)
0929 10:58:21.394807 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:21.394840 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":75,"threads_started":1}
I20260929 10:58:21.478262 28121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:21.478016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.194:37577)
0929 10:58:21.478155 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:21.478192 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":94,"threads_started":1}
I20260929 10:58:21.603885 28122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:21.603667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.195:43391)
0929 10:58:21.603803 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:21.603843 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":75,"threads_started":1}
I20260929 10:58:21.785456 28123 raft_consensus.cc:482] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:58:21.785553 28123 raft_consensus.cc:504] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f941f11113449088d27274866fc18d2" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38929 } } peers { permanent_uuid: "87a7fa2b136444c389cf89f4114fb9b4" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36489 } } peers { permanent_uuid: "79a93325cbb74798842ddee222b050ee" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 32973 } }
I20260929 10:58:21.785715 28123 leader_election.cc:288] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4f941f11113449088d27274866fc18d2 (127.10.238.194:38929), 87a7fa2b136444c389cf89f4114fb9b4 (127.10.238.195:36489)
W20260929 10:58:21.786085 27709 leader_election.cc:341] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4f941f11113449088d27274866fc18d2 (127.10.238.194:38929): Not found: Tablet not found: da6d9eabd9ec4a49927c636902746e7f
W20260929 10:58:21.786146 27707 leader_election.cc:341] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87a7fa2b136444c389cf89f4114fb9b4 (127.10.238.195:36489): Not found: Tablet not found: da6d9eabd9ec4a49927c636902746e7f
I20260929 10:58:21.786173 27709 leader_election.cc:302] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 79a93325cbb74798842ddee222b050ee; no voters: 4f941f11113449088d27274866fc18d2, 87a7fa2b136444c389cf89f4114fb9b4
I20260929 10:58:21.786253 28123 raft_consensus.cc:2726] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 10:58:22.395556 28124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:22.395341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.193:45849)
0929 10:58:22.395475 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:22.395509 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260929 10:58:22.478991 28125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:22.478746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.194:59733)
0929 10:58:22.478896 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:22.478937 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":65,"threads_started":1}
I20260929 10:58:22.604568 28126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:22.604333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.195:33011)
0929 10:58:22.604486 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:22.604524 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":73,"threads_started":1}
I20260929 10:58:23.396239 28127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:23.395975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.193:43261)
0929 10:58:23.396127 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:23.396168 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":64,"threads_started":1}
I20260929 10:58:23.479719 28128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:23.479491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.194:39927)
0929 10:58:23.479620 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:23.479656 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":70,"threads_started":1}
I20260929 10:58:23.605229 28129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:23.605006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.195:59519)
0929 10:58:23.605149 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:23.605186 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":70,"threads_started":1}
I20260929 10:58:23.833349 28130 raft_consensus.cc:482] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:58:23.833436 28130 raft_consensus.cc:504] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f941f11113449088d27274866fc18d2" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38929 } } peers { permanent_uuid: "87a7fa2b136444c389cf89f4114fb9b4" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36489 } } peers { permanent_uuid: "79a93325cbb74798842ddee222b050ee" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 32973 } }
I20260929 10:58:23.833632 28130 leader_election.cc:288] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4f941f11113449088d27274866fc18d2 (127.10.238.194:38929), 87a7fa2b136444c389cf89f4114fb9b4 (127.10.238.195:36489)
W20260929 10:58:23.834127 27709 leader_election.cc:341] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4f941f11113449088d27274866fc18d2 (127.10.238.194:38929): Not found: Tablet not found: da6d9eabd9ec4a49927c636902746e7f
W20260929 10:58:23.834218 27707 leader_election.cc:341] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87a7fa2b136444c389cf89f4114fb9b4 (127.10.238.195:36489): Not found: Tablet not found: da6d9eabd9ec4a49927c636902746e7f
I20260929 10:58:23.834236 27707 leader_election.cc:302] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 79a93325cbb74798842ddee222b050ee; no voters: 4f941f11113449088d27274866fc18d2, 87a7fa2b136444c389cf89f4114fb9b4
I20260929 10:58:23.834333 28130 raft_consensus.cc:2726] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 10:58:24.396960 28131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:24.396757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.193:40573)
0929 10:58:24.396878 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:24.396911 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":61,"threads_started":1}
I20260929 10:58:24.480365 28132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:24.480164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.194:60767)
0929 10:58:24.480279 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:24.480320 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":61,"threads_started":1}
I20260929 10:58:24.605909 28133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:24.605697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.195:33597)
0929 10:58:24.605819 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:24.605855 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":58,"threads_started":1}
I20260929 10:58:25.397729 28134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:25.397476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.193:50347)
0929 10:58:25.397634 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:25.397679 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":65,"threads_started":1}
I20260929 10:58:25.481119 28135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:25.480914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.194:56497)
0929 10:58:25.481045 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:25.481077 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":61,"threads_started":1}
I20260929 10:58:25.606564 28136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:25.606343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.195:38237)
0929 10:58:25.606486 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:25.606521 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":74,"threads_started":1}
I20260929 10:58:26.398447 28137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:26.398226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.193:45257)
0929 10:58:26.398359 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:26.398400 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":65,"threads_started":1}
I20260929 10:58:26.481814 28138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:26.481608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.194:37295)
0929 10:58:26.481732 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:26.481769 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":59,"threads_started":1}
I20260929 10:58:26.607203 28139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:26.606994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.195:44103)
0929 10:58:26.607124 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:26.607161 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":72,"threads_started":1}
I20260929 10:58:26.847860 28140 raft_consensus.cc:482] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:58:26.847985 28140 raft_consensus.cc:504] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f941f11113449088d27274866fc18d2" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38929 } } peers { permanent_uuid: "87a7fa2b136444c389cf89f4114fb9b4" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 36489 } } peers { permanent_uuid: "79a93325cbb74798842ddee222b050ee" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 32973 } }
I20260929 10:58:26.848198 28140 leader_election.cc:288] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4f941f11113449088d27274866fc18d2 (127.10.238.194:38929), 87a7fa2b136444c389cf89f4114fb9b4 (127.10.238.195:36489)
W20260929 10:58:26.848567 27709 leader_election.cc:341] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4f941f11113449088d27274866fc18d2 (127.10.238.194:38929): Not found: Tablet not found: da6d9eabd9ec4a49927c636902746e7f
W20260929 10:58:26.848662 27707 leader_election.cc:341] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87a7fa2b136444c389cf89f4114fb9b4 (127.10.238.195:36489): Not found: Tablet not found: da6d9eabd9ec4a49927c636902746e7f
I20260929 10:58:26.848735 27707 leader_election.cc:302] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 79a93325cbb74798842ddee222b050ee; no voters: 4f941f11113449088d27274866fc18d2, 87a7fa2b136444c389cf89f4114fb9b4
I20260929 10:58:26.848805 28140 raft_consensus.cc:2726] T da6d9eabd9ec4a49927c636902746e7f P 79a93325cbb74798842ddee222b050ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 10:58:27.399209 28141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:27.398965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.193:60333)
0929 10:58:27.399111 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:27.399153 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":62,"threads_started":1}
I20260929 10:58:27.482529 28142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:27.482308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.194:38215)
0929 10:58:27.482449 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:27.482484 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":76,"threads_started":1}
I20260929 10:58:27.607249 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 27656
I20260929 10:58:27.608723 28143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:27.608012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:39121 (local address 127.10.238.195:59239)
0929 10:58:27.608635 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:27.608676 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.254:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":62,"threads_started":1}
I20260929 10:58:27.612663 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 27824
I20260929 10:58:27.617586 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 27957
2026-09-29T10:58:27Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58643 ms)
[----------] 30 tests from SecurityITest (233116 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:58:27 dist-test-slave-8nrt krb5kdc[28146](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:58:27 dist-test-slave-8nrt krb5kdc[28146](info): set up 2 sockets
Sep 29 10:58:27 dist-test-slave-8nrt krb5kdc[28146](info): commencing operation
krb5kdc: starting...
W20260929 10:58:28.657150 11195 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Sep 29 10:58:28 dist-test-slave-8nrt krb5kdc[28146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:58:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:58:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
I20260929 10:58:28.779892 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:34325
--webserver_interface=127.10.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:36379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:34325
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:58:28.866194 28159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:58:28.866431 28159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:58:28.866469 28159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:58:28.867619 28159 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:58:28.867686 28159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:58:28.867713 28159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:58:28.867750 28159 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:58:28.867774 28159 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:58:28.869241 28159 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:36379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:34325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:34325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:58:28.869562 28159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:58:28.869788 28159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:58:28.875157 28164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:28.875146 28167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:28.875252 28159 server_base.cc:1034] running on GCE node
W20260929 10:58:28.875154 28165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:28.875692 28159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:58:28.876037 28159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:58:28.877219 28159 hybrid_clock.cc:648] HybridClock initialized: now 1790679508877211 us; error 37 us; skew 500 ppm
Sep 29 10:58:28 dist-test-slave-8nrt krb5kdc[28146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679508, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:58:28.883550 28159 init.cc:377] Logged in from keytab as kudu/127.10.238.254@KRBTEST.COM (short username kudu)
I20260929 10:58:28.884007 28159 webserver.cc:460] Webserver started at http://127.10.238.254:43281/ using document root <none> and password file <none>
I20260929 10:58:28.884263 28159 fs_manager.cc:359] Metadata directory not provided
I20260929 10:58:28.884334 28159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:58:28.884481 28159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:58:28.885633 28159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/master-0/data/instance:
uuid: "f481b8f389ae42728c77739eb8d2b543"
format_stamp: "Formatted at 2026-09-29 10:58:28 on dist-test-slave-8nrt"
server_key: "0a734776084806cde42d73832315613f"
server_key_iv: "ffb8871f62b6185584d538752e1c0e43"
server_key_version: "encryptionkey@0"
I20260929 10:58:28.885996 28159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/master-0/wal/instance:
uuid: "f481b8f389ae42728c77739eb8d2b543"
format_stamp: "Formatted at 2026-09-29 10:58:28 on dist-test-slave-8nrt"
server_key: "0a734776084806cde42d73832315613f"
server_key_iv: "ffb8871f62b6185584d538752e1c0e43"
server_key_version: "encryptionkey@0"
I20260929 10:58:28.887323 28159 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260929 10:58:28.888093 28174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:28.888319 28159 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:58:28.888429 28159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "f481b8f389ae42728c77739eb8d2b543"
format_stamp: "Formatted at 2026-09-29 10:58:28 on dist-test-slave-8nrt"
server_key: "0a734776084806cde42d73832315613f"
server_key_iv: "ffb8871f62b6185584d538752e1c0e43"
server_key_version: "encryptionkey@0"
I20260929 10:58:28.888512 28159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:58:28.908180 28159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:58:28.910240 28159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:58:28.910416 28159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:58:28.914605 28159 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.254:34325
I20260929 10:58:28.914717 28226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.254:34325 every 8 connection(s)
I20260929 10:58:28.915043 28159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:58:28.915989 28227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:58:28.918550 28227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f481b8f389ae42728c77739eb8d2b543: Bootstrap starting.
I20260929 10:58:28.919224 28227 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f481b8f389ae42728c77739eb8d2b543: Neither blocks nor log segments found. Creating new log.
I20260929 10:58:28.919560 28227 log.cc:824] T 00000000000000000000000000000000 P f481b8f389ae42728c77739eb8d2b543: Log is configured to *not* fsync() on all Append() calls
I20260929 10:58:28.920384 28227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f481b8f389ae42728c77739eb8d2b543: No bootstrap required, opened a new log
I20260929 10:58:28.922186 28227 raft_consensus.cc:348] T 00000000000000000000000000000000 P f481b8f389ae42728c77739eb8d2b543 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f481b8f389ae42728c77739eb8d2b543" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 34325 } }
I20260929 10:58:28.922329 28227 raft_consensus.cc:374] T 00000000000000000000000000000000 P f481b8f389ae42728c77739eb8d2b543 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:58:28.922351 28227 raft_consensus.cc:729] T 00000000000000000000000000000000 P f481b8f389ae42728c77739eb8d2b543 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f481b8f389ae42728c77739eb8d2b543, State: Initialized, Role: FOLLOWER
I20260929 10:58:28.922510 28227 consensus_queue.cc:260] T 00000000000000000000000000000000 P f481b8f389ae42728c77739eb8d2b543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f481b8f389ae42728c77739eb8d2b543" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 34325 } }
I20260929 10:58:28.922587 28227 raft_consensus.cc:388] T 00000000000000000000000000000000 P f481b8f389ae42728c77739eb8d2b543 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:58:28.922609 28227 raft_consensus.cc:482] T 00000000000000000000000000000000 P f481b8f389ae42728c77739eb8d2b543 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:58:28.922689 28227 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f481b8f389ae42728c77739eb8d2b543 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:58:28.923249 28227 raft_consensus.cc:504] T 00000000000000000000000000000000 P f481b8f389ae42728c77739eb8d2b543 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f481b8f389ae42728c77739eb8d2b543" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 34325 } }
I20260929 10:58:28.923365 28227 leader_election.cc:302] T 00000000000000000000000000000000 P f481b8f389ae42728c77739eb8d2b543 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f481b8f389ae42728c77739eb8d2b543; no voters:
I20260929 10:58:28.923564 28227 leader_election.cc:288] T 00000000000000000000000000000000 P f481b8f389ae42728c77739eb8d2b543 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:58:28.923684 28230 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f481b8f389ae42728c77739eb8d2b543 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:58:28.923913 28230 raft_consensus.cc:686] T 00000000000000000000000000000000 P f481b8f389ae42728c77739eb8d2b543 [term 1 LEADER]: Becoming Leader. State: Replica: f481b8f389ae42728c77739eb8d2b543, State: Running, Role: LEADER
I20260929 10:58:28.923952 28227 sys_catalog.cc:564] T 00000000000000000000000000000000 P f481b8f389ae42728c77739eb8d2b543 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:58:28.924008 28230 consensus_queue.cc:237] T 00000000000000000000000000000000 P f481b8f389ae42728c77739eb8d2b543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f481b8f389ae42728c77739eb8d2b543" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 34325 } }
I20260929 10:58:28.924384 28232 sys_catalog.cc:455] T 00000000000000000000000000000000 P f481b8f389ae42728c77739eb8d2b543 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f481b8f389ae42728c77739eb8d2b543. Latest consensus state: current_term: 1 leader_uuid: "f481b8f389ae42728c77739eb8d2b543" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f481b8f389ae42728c77739eb8d2b543" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 34325 } } }
I20260929 10:58:28.924520 28231 sys_catalog.cc:455] T 00000000000000000000000000000000 P f481b8f389ae42728c77739eb8d2b543 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f481b8f389ae42728c77739eb8d2b543" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f481b8f389ae42728c77739eb8d2b543" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 34325 } } }
I20260929 10:58:28.924612 28231 sys_catalog.cc:458] T 00000000000000000000000000000000 P f481b8f389ae42728c77739eb8d2b543 [sys.catalog]: This master's current role is: LEADER
I20260929 10:58:28.924543 28232 sys_catalog.cc:458] T 00000000000000000000000000000000 P f481b8f389ae42728c77739eb8d2b543 [sys.catalog]: This master's current role is: LEADER
I20260929 10:58:28.924841 28239 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:58:28.924916 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 28159
I20260929 10:58:28.925036 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/master-0/wal/instance
I20260929 10:58:28.925333 11195 external_mini_cluster.cc:1428] Setting key 20596d5c22622ce7ce0759a9093f4b15
I20260929 10:58:28.925436 28239 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:58:28.931005 28239 catalog_manager.cc:1347] Generated new cluster ID: e6d658046c6d423290bbd8c4a3fc1b59
I20260929 10:58:28.931088 28239 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 10:58:28 dist-test-slave-8nrt krb5kdc[28146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:58:28.936272 28249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:28.925912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44026 (local address 127.10.238.254:34325)
0929 10:58:28.926073 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:28.926083 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:28.926176 (+ 93us) server_negotiation.cc:407] Connection header received
0929 10:58:28.926336 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:28.926343 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:28.926389 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:28.926525 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:28.927192 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:28.928102 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:28.928988 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:28.929480 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:28.929616 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:28.932492 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:28.932496 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:28.932497 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:58:28.932509 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:28.934586 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:28.935011 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:28.935014 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:28.935015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:28.935138 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:28.935348 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:28.935349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:28.935349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:28.935677 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:28.935831 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:28.935912 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 10:58:28.936051 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":82,"threads_started":1}
I20260929 10:58:28.939857 28239 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:58:28.940577 28239 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:58:28.948930 28239 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f481b8f389ae42728c77739eb8d2b543: Generated new TSK 0
I20260929 10:58:28.949136 28239 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.238.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab.
Entry for principal kudu/127.10.238.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab.
I20260929 10:58:28.980412 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.193:0
--local_ip_for_outbound_sockets=127.10.238.193
--webserver_interface=127.10.238.193
--webserver_port=0
--tserver_master_addrs=127.10.238.254:34325
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab
--principal=kudu/127.10.238.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:36379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:58:29.063287 28253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:58:29.063508 28253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:58:29.063534 28253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:58:29.064736 28253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:58:29.064780 28253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:58:29.064806 28253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.193
I20260929 10:58:29.066365 28253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:36379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab
--principal=kudu/127.10.238.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.238.193
--webserver_port=0
--tserver_master_addrs=127.10.238.254:34325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.193
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:58:29.066618 28253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:58:29.066861 28253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:58:29.072000 28261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:29.072166 28253 server_base.cc:1034] running on GCE node
W20260929 10:58:29.072017 28259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:29.071970 28258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:29.072489 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:58:29.072745 28253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:58:29.073895 28253 hybrid_clock.cc:648] HybridClock initialized: now 1790679509073874 us; error 35 us; skew 500 ppm
Sep 29 10:58:29 dist-test-slave-8nrt krb5kdc[28146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679509, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:58:29.076699 28253 init.cc:377] Logged in from keytab as kudu/127.10.238.193@KRBTEST.COM (short username kudu)
I20260929 10:58:29.077200 28253 webserver.cc:460] Webserver started at http://127.10.238.193:39729/ using document root <none> and password file <none>
I20260929 10:58:29.077485 28253 fs_manager.cc:359] Metadata directory not provided
I20260929 10:58:29.077535 28253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:58:29.077694 28253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:58:29.078792 28253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/ts-0/data/instance:
uuid: "7d9d31444489446fad9dcfe06ebcb0d7"
format_stamp: "Formatted at 2026-09-29 10:58:29 on dist-test-slave-8nrt"
server_key: "6d3ad89120882f384b3ecdd6ea19dd4f"
server_key_iv: "67a4845cd9fe8fb691e036fe63f474d3"
server_key_version: "encryptionkey@0"
I20260929 10:58:29.079166 28253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/ts-0/wal/instance:
uuid: "7d9d31444489446fad9dcfe06ebcb0d7"
format_stamp: "Formatted at 2026-09-29 10:58:29 on dist-test-slave-8nrt"
server_key: "6d3ad89120882f384b3ecdd6ea19dd4f"
server_key_iv: "67a4845cd9fe8fb691e036fe63f474d3"
server_key_version: "encryptionkey@0"
I20260929 10:58:29.080543 28253 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 10:58:29.081416 28268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:29.081669 28253 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:58:29.081743 28253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/ts-0/wal
uuid: "7d9d31444489446fad9dcfe06ebcb0d7"
format_stamp: "Formatted at 2026-09-29 10:58:29 on dist-test-slave-8nrt"
server_key: "6d3ad89120882f384b3ecdd6ea19dd4f"
server_key_iv: "67a4845cd9fe8fb691e036fe63f474d3"
server_key_version: "encryptionkey@0"
I20260929 10:58:29.081810 28253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:58:29.090171 28253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:58:29.092293 28253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:58:29.092447 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:58:29.092679 28253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:58:29.092998 28253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:58:29.093031 28253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:29.093075 28253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:58:29.093104 28253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:29.099263 28253 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.193:36095
I20260929 10:58:29.099314 28381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.193:36095 every 8 connection(s)
I20260929 10:58:29.099627 28253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
Sep 29 10:58:29 dist-test-slave-8nrt krb5kdc[28146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679509, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.193@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:58:29.105603 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 28253
I20260929 10:58:29.105736 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/ts-0/wal/instance
I20260929 10:58:29.105954 11195 external_mini_cluster.cc:1428] Setting key 4710f2bb0aa205126114e7fcc033f765
I20260929 10:58:29.109225 28384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:29.100336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:34325 (local address 127.10.238.193:57531)
0929 10:58:29.100509 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:29.100550 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 10:58:29.101038 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:58:29.101238 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:58:29.101241 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:58:29.101848 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:58:29.102041 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:29.102046 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:29.102745 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:29.102746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:29.103538 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:29.103539 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:29.103884 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:29.103885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:29.103998 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:29.104501 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:58:29.104509 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:58:29.106056 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:58:29.107774 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:58:29.107782 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:58:29.107784 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:58:29.108099 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:58:29.108349 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:58:29.108350 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:58:29.108350 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:58:29.108450 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:58:29.108736 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:58:29.108741 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:58:29.108854 (+ 113us) client_negotiation.cc:769] Sending connection context
0929 10:58:29.108937 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 10:58:29.109055 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":104,"threads_started":1}
I20260929 10:58:29.109222 28249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:29.100371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:57531 (local address 127.10.238.254:34325)
0929 10:58:29.100429 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:29.100431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:29.100641 (+ 210us) server_negotiation.cc:407] Connection header received
0929 10:58:29.101118 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:29.101118 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:29.101130 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:29.101185 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:29.102148 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:29.102655 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:29.103612 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:29.103808 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:29.103859 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:29.106167 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:29.106169 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:29.106169 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:58:29.106178 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:29.107687 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:29.108198 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:29.108199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:29.108200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:29.108275 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:29.108523 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:29.108524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:29.108525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:29.108670 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:29.108740 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:29.108985 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 10:58:29.109086 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 10:58:29.109926 28382 heartbeater.cc:347] Connected to a master server at 127.10.238.254:34325
I20260929 10:58:29.110041 28382 heartbeater.cc:464] Registering TS with master...
I20260929 10:58:29.110215 28382 heartbeater.cc:511] Master 127.10.238.254:34325 requested a full tablet report, sending...
I20260929 10:58:29.110889 28191 ts_manager.cc:194] Registered new tserver with Master: 7d9d31444489446fad9dcfe06ebcb0d7 (127.10.238.193:36095)
I20260929 10:58:29.111968 28191 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.193@KRBTEST.COM'} at 127.10.238.193:57531
WARNING: no policy specified for kudu/127.10.238.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab.
Entry for principal kudu/127.10.238.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab.
I20260929 10:58:29.132148 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.194:0
--local_ip_for_outbound_sockets=127.10.238.194
--webserver_interface=127.10.238.194
--webserver_port=0
--tserver_master_addrs=127.10.238.254:34325
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab
--principal=kudu/127.10.238.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:36379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:58:29.215291 28387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:58:29.215499 28387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:58:29.215523 28387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:58:29.216737 28387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:58:29.216781 28387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:58:29.216809 28387 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.194
I20260929 10:58:29.218336 28387 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:36379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab
--principal=kudu/127.10.238.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.238.194
--webserver_port=0
--tserver_master_addrs=127.10.238.254:34325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.194
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:58:29.218581 28387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:58:29.218837 28387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:58:29.223857 28392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:29.223968 28395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:29.223964 28393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:29.224068 28387 server_base.cc:1034] running on GCE node
I20260929 10:58:29.224329 28387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:58:29.224591 28387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:58:29.225776 28387 hybrid_clock.cc:648] HybridClock initialized: now 1790679509225757 us; error 64 us; skew 500 ppm
Sep 29 10:58:29 dist-test-slave-8nrt krb5kdc[28146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679509, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:58:29.228171 28387 init.cc:377] Logged in from keytab as kudu/127.10.238.194@KRBTEST.COM (short username kudu)
I20260929 10:58:29.228608 28387 webserver.cc:460] Webserver started at http://127.10.238.194:41207/ using document root <none> and password file <none>
I20260929 10:58:29.228853 28387 fs_manager.cc:359] Metadata directory not provided
I20260929 10:58:29.228924 28387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:58:29.229074 28387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:58:29.230178 28387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/ts-1/data/instance:
uuid: "cbee193e29874985890bcf6bc4bcb668"
format_stamp: "Formatted at 2026-09-29 10:58:29 on dist-test-slave-8nrt"
server_key: "71a4ca227aebdddc596f3dcc8c201f80"
server_key_iv: "de6bcce4c2cd83800ec65c57801460a5"
server_key_version: "encryptionkey@0"
I20260929 10:58:29.230607 28387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/ts-1/wal/instance:
uuid: "cbee193e29874985890bcf6bc4bcb668"
format_stamp: "Formatted at 2026-09-29 10:58:29 on dist-test-slave-8nrt"
server_key: "71a4ca227aebdddc596f3dcc8c201f80"
server_key_iv: "de6bcce4c2cd83800ec65c57801460a5"
server_key_version: "encryptionkey@0"
I20260929 10:58:29.231946 28387 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 10:58:29.232659 28402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:29.232837 28387 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:58:29.232939 28387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/ts-1/wal
uuid: "cbee193e29874985890bcf6bc4bcb668"
format_stamp: "Formatted at 2026-09-29 10:58:29 on dist-test-slave-8nrt"
server_key: "71a4ca227aebdddc596f3dcc8c201f80"
server_key_iv: "de6bcce4c2cd83800ec65c57801460a5"
server_key_version: "encryptionkey@0"
I20260929 10:58:29.233019 28387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:58:29.247105 28387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:58:29.249069 28387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:58:29.249301 28387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:58:29.249571 28387 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:58:29.249913 28387 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:58:29.249971 28387 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:29.250013 28387 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:58:29.250041 28387 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:29.256282 28387 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.194:36469
I20260929 10:58:29.256337 28515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.194:36469 every 8 connection(s)
I20260929 10:58:29.256771 28387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
I20260929 10:58:29.256994 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 28387
I20260929 10:58:29.257128 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/ts-1/wal/instance
I20260929 10:58:29.257377 11195 external_mini_cluster.cc:1428] Setting key 5b8ee00850c1f7f6734517e6a60a35aa
Sep 29 10:58:29 dist-test-slave-8nrt krb5kdc[28146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679509, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.194@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.238.195@KRBTEST.COM; defaulting to no policy
I20260929 10:58:29.266938 28249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:29.257671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:49935 (local address 127.10.238.254:34325)
0929 10:58:29.257752 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:29.257755 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:29.257991 (+ 236us) server_negotiation.cc:407] Connection header received
0929 10:58:29.258530 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:29.258531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:29.258543 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:29.258605 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:29.259532 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:29.260038 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:29.261164 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:29.261464 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:29.261553 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:29.263987 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:29.263988 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:29.263989 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:58:29.263996 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:29.265497 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:29.265929 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:29.265930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:29.265931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:29.265988 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:29.266273 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:29.266274 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:29.266274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:29.266393 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:29.266472 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:29.266758 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 10:58:29.266845 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 10:58:29.267269 28518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:29.257657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:34325 (local address 127.10.238.194:49935)
0929 10:58:29.257862 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:29.257901 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 10:58:29.258436 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:58:29.258659 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:58:29.258662 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:58:29.259230 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:58:29.259420 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:29.259425 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:29.260140 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:29.260141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:29.261041 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:29.261044 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:29.261565 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:29.261566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:29.261697 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:29.262228 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:58:29.262236 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:58:29.263885 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:58:29.265585 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:58:29.265592 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:58:29.265594 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:58:29.265830 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:58:29.266059 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:58:29.266060 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:58:29.266060 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:58:29.266201 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:58:29.266463 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:58:29.266470 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:58:29.266614 (+ 144us) client_negotiation.cc:769] Sending connection context
0929 10:58:29.266726 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 10:58:29.266885 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":106,"threads_started":1}
I20260929 10:58:29.267768 28516 heartbeater.cc:347] Connected to a master server at 127.10.238.254:34325
I20260929 10:58:29.267890 28516 heartbeater.cc:464] Registering TS with master...
I20260929 10:58:29.268071 28516 heartbeater.cc:511] Master 127.10.238.254:34325 requested a full tablet report, sending...
I20260929 10:58:29.268481 28191 ts_manager.cc:194] Registered new tserver with Master: cbee193e29874985890bcf6bc4bcb668 (127.10.238.194:36469)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.195@KRBTEST.COM" created.
I20260929 10:58:29.268942 28191 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.194@KRBTEST.COM'} at 127.10.238.194:49935
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab.
Entry for principal kudu/127.10.238.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab.
I20260929 10:58:29.284164 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.195:0
--local_ip_for_outbound_sockets=127.10.238.195
--webserver_interface=127.10.238.195
--webserver_port=0
--tserver_master_addrs=127.10.238.254:34325
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab
--principal=kudu/127.10.238.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:36379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:58:29.369189 28521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:58:29.369418 28521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:58:29.369442 28521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:58:29.370527 28521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:58:29.370561 28521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:58:29.370587 28521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.195
I20260929 10:58:29.371959 28521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:36379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab
--principal=kudu/127.10.238.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.238.195
--webserver_port=0
--tserver_master_addrs=127.10.238.254:34325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.195
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:58:29.372148 28521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:58:29.372320 28521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:58:29.377492 28526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:29.377467 28527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:29.377548 28521 server_base.cc:1034] running on GCE node
W20260929 10:58:29.377462 28529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:29.377833 28521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:58:29.378111 28521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:58:29.379285 28521 hybrid_clock.cc:648] HybridClock initialized: now 1790679509379265 us; error 57 us; skew 500 ppm
Sep 29 10:58:29 dist-test-slave-8nrt krb5kdc[28146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679509, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:58:29.381973 28521 init.cc:377] Logged in from keytab as kudu/127.10.238.195@KRBTEST.COM (short username kudu)
I20260929 10:58:29.382449 28521 webserver.cc:460] Webserver started at http://127.10.238.195:43617/ using document root <none> and password file <none>
I20260929 10:58:29.382696 28521 fs_manager.cc:359] Metadata directory not provided
I20260929 10:58:29.382771 28521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:58:29.382920 28521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:58:29.383973 28521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/ts-2/data/instance:
uuid: "90773ff7f66d457498922c8b38714fa4"
format_stamp: "Formatted at 2026-09-29 10:58:29 on dist-test-slave-8nrt"
server_key: "1cc51558f4881af547400dc45e8c101f"
server_key_iv: "0de33d7ad3fce8214c667ff2de5ead06"
server_key_version: "encryptionkey@0"
I20260929 10:58:29.384356 28521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/ts-2/wal/instance:
uuid: "90773ff7f66d457498922c8b38714fa4"
format_stamp: "Formatted at 2026-09-29 10:58:29 on dist-test-slave-8nrt"
server_key: "1cc51558f4881af547400dc45e8c101f"
server_key_iv: "0de33d7ad3fce8214c667ff2de5ead06"
server_key_version: "encryptionkey@0"
I20260929 10:58:29.385882 28521 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 10:58:29.386610 28536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:29.386781 28521 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:58:29.386879 28521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/ts-2/wal
uuid: "90773ff7f66d457498922c8b38714fa4"
format_stamp: "Formatted at 2026-09-29 10:58:29 on dist-test-slave-8nrt"
server_key: "1cc51558f4881af547400dc45e8c101f"
server_key_iv: "0de33d7ad3fce8214c667ff2de5ead06"
server_key_version: "encryptionkey@0"
I20260929 10:58:29.386958 28521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:58:29.400359 28521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:58:29.402855 28521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:58:29.403028 28521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:58:29.403278 28521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:58:29.403601 28521 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:58:29.403664 28521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:29.403736 28521 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:58:29.403769 28521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:29.410755 28521 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.195:37233
I20260929 10:58:29.410808 28649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.195:37233 every 8 connection(s)
I20260929 10:58:29.411123 28521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
Sep 29 10:58:29 dist-test-slave-8nrt krb5kdc[28146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679509, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.195@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:58:29.419323 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 28521
I20260929 10:58:29.419468 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/ts-2/wal/instance
I20260929 10:58:29.419714 11195 external_mini_cluster.cc:1428] Setting key 36ef3f72dea230df6d6a27ee74a63a35
I20260929 10:58:29.421082 28249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:29.411928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:51453 (local address 127.10.238.254:34325)
0929 10:58:29.411998 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:29.412000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:29.412214 (+ 214us) server_negotiation.cc:407] Connection header received
0929 10:58:29.412768 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:29.412770 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:29.412783 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:29.412824 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:29.413870 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:29.414378 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:29.415406 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:29.415601 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:29.415690 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:29.417995 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:29.417996 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:29.417997 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:58:29.418004 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:29.419420 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:29.419944 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:29.419945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:29.419946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:29.420011 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:29.420351 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:29.420352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:29.420352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:29.420528 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:29.420594 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:29.420953 (+ 359us) server_negotiation.cc:299] Negotiation successful
0929 10:58:29.421020 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 10:58:29.421263 28652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:29.411895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:34325 (local address 127.10.238.195:51453)
0929 10:58:29.412079 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:29.412124 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 10:58:29.412651 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:58:29.412894 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:58:29.412898 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:58:29.413552 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:58:29.413757 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:29.413762 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:29.414480 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:29.414481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:29.415297 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:29.415299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:29.415720 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:29.415722 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:29.415850 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:29.416403 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:58:29.416411 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:58:29.417895 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:58:29.419525 (+ 1630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:58:29.419532 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:58:29.419535 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:58:29.419848 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:58:29.420082 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:58:29.420083 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:58:29.420083 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:58:29.420256 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:58:29.420598 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:58:29.420604 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:58:29.420794 (+ 190us) client_negotiation.cc:769] Sending connection context
0929 10:58:29.420911 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 10:58:29.421053 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":97,"threads_started":1}
I20260929 10:58:29.421710 28650 heartbeater.cc:347] Connected to a master server at 127.10.238.254:34325
I20260929 10:58:29.421808 28650 heartbeater.cc:464] Registering TS with master...
I20260929 10:58:29.421949 28650 heartbeater.cc:511] Master 127.10.238.254:34325 requested a full tablet report, sending...
I20260929 10:58:29.422277 28191 ts_manager.cc:194] Registered new tserver with Master: 90773ff7f66d457498922c8b38714fa4 (127.10.238.195:37233)
I20260929 10:58:29.422610 28191 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.195@KRBTEST.COM'} at 127.10.238.195:51453
I20260929 10:58:29.431591 11195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:58:29.439126 28249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:29.433142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44030 (local address 127.10.238.254:34325)
0929 10:58:29.433212 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:29.433214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:29.433352 (+ 138us) server_negotiation.cc:407] Connection header received
0929 10:58:29.433400 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:29.433401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:29.433409 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:29.433465 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:29.434083 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:29.434573 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:29.435285 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:29.435471 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:29.435526 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:29.436826 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:29.436827 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:29.436827 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:58:29.436834 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:29.438146 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:29.438472 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:29.438472 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:29.438473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:29.438525 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:29.438688 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:29.438689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:29.438689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:29.438837 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:29.438915 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:29.438999 (+ 84us) server_negotiation.cc:299] Negotiation successful
0929 10:58:29.439061 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 10:58:29.440927 28191 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44030:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:58:29.442756 28191 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:58:29.450004 28658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:29.445493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.195:37233 (local address 127.0.0.1:50150)
0929 10:58:29.445953 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:29.445962 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 10:58:29.446009 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:58:29.446276 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:58:29.446279 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:58:29.446282 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:58:29.446360 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:29.446362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:29.447766 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:29.447767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:29.448815 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:29.448817 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:29.449584 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:29.449585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:29.449722 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:29.449736 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 10:58:29.449835 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 10:58:29.449908 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":26,"thread_start_us":72,"threads_started":1}
I20260929 10:58:29.450029 28663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:29.445608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50150 (local address 127.10.238.195:37233)
0929 10:58:29.445793 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:29.445799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:29.446077 (+ 278us) server_negotiation.cc:407] Connection header received
0929 10:58:29.446129 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:29.446131 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:29.446160 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:29.446247 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:58:29.446459 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:29.447647 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:29.448900 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:29.449495 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:29.449584 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:29.449640 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:29.449888 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 10:58:29.449936 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":101,"threads_started":1}
I20260929 10:58:29.450016 28661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:29.445287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.194:36469 (local address 127.0.0.1:53552)
0929 10:58:29.445808 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:29.445818 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 10:58:29.445871 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:58:29.446597 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:58:29.446599 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:58:29.446601 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:58:29.446641 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:29.446643 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:29.447494 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:29.447495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:29.448762 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:29.448765 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:29.449791 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:29.449791 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:29.449855 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:29.449859 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 10:58:29.449910 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 10:58:29.449949 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"mutex_wait_us":5,"thread_start_us":56,"threads_started":1}
I20260929 10:58:29.450270 28660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:29.445400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53552 (local address 127.10.238.194:36469)
0929 10:58:29.446019 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:29.446024 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:29.446033 (+ 9us) server_negotiation.cc:407] Connection header received
0929 10:58:29.446092 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:29.446095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:29.446111 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:29.446594 (+ 483us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:58:29.446699 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:29.447384 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:29.448836 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:29.449704 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:29.449775 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:29.449824 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:29.450057 (+ 233us) server_negotiation.cc:299] Negotiation successful
0929 10:58:29.450208 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":158,"threads_started":1}
I20260929 10:58:29.450683 28659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:29.445287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:36095 (local address 127.0.0.1:54170)
0929 10:58:29.445684 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:29.445730 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 10:58:29.445835 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:58:29.446459 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:58:29.446460 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:58:29.446462 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:58:29.446521 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:29.446522 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:29.447803 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:29.447804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:29.448659 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:29.448662 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:29.449670 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:29.449671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:29.449754 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:29.449759 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 10:58:29.449828 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 10:58:29.449901 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":65,"threads_started":1}
I20260929 10:58:29.450912 28662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:29.445491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54170 (local address 127.10.238.193:36095)
0929 10:58:29.446263 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:29.446268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:29.446277 (+ 9us) server_negotiation.cc:407] Connection header received
0929 10:58:29.446314 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:29.446316 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:29.446330 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:29.446418 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:58:29.446620 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:29.447720 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:29.448761 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:29.449506 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:29.449580 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:29.449635 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:29.449949 (+ 314us) server_negotiation.cc:299] Negotiation successful
0929 10:58:29.449994 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":731,"thread_start_us":89,"threads_started":1}
I20260929 10:58:29.450982 28450 tablet_service.cc:1467] Processing CreateTablet for tablet f5274ab4f7ad40f5838c325084064677 (DEFAULT_TABLE table=test-table [id=a8554c4a667f475b8cbb0570355da174]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:58:29.451032 28584 tablet_service.cc:1467] Processing CreateTablet for tablet f5274ab4f7ad40f5838c325084064677 (DEFAULT_TABLE table=test-table [id=a8554c4a667f475b8cbb0570355da174]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:58:29.451388 28584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5274ab4f7ad40f5838c325084064677. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:58:29.451388 28450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5274ab4f7ad40f5838c325084064677. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:58:29.451508 28316 tablet_service.cc:1467] Processing CreateTablet for tablet f5274ab4f7ad40f5838c325084064677 (DEFAULT_TABLE table=test-table [id=a8554c4a667f475b8cbb0570355da174]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:58:29.451746 28316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5274ab4f7ad40f5838c325084064677. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:58:29.454617 28664 tablet_bootstrap.cc:492] T f5274ab4f7ad40f5838c325084064677 P 7d9d31444489446fad9dcfe06ebcb0d7: Bootstrap starting.
I20260929 10:58:29.454958 28665 tablet_bootstrap.cc:492] T f5274ab4f7ad40f5838c325084064677 P 90773ff7f66d457498922c8b38714fa4: Bootstrap starting.
I20260929 10:58:29.455049 28666 tablet_bootstrap.cc:492] T f5274ab4f7ad40f5838c325084064677 P cbee193e29874985890bcf6bc4bcb668: Bootstrap starting.
I20260929 10:58:29.455448 28664 tablet_bootstrap.cc:654] T f5274ab4f7ad40f5838c325084064677 P 7d9d31444489446fad9dcfe06ebcb0d7: Neither blocks nor log segments found. Creating new log.
I20260929 10:58:29.455809 28666 tablet_bootstrap.cc:654] T f5274ab4f7ad40f5838c325084064677 P cbee193e29874985890bcf6bc4bcb668: Neither blocks nor log segments found. Creating new log.
I20260929 10:58:29.455809 28665 tablet_bootstrap.cc:654] T f5274ab4f7ad40f5838c325084064677 P 90773ff7f66d457498922c8b38714fa4: Neither blocks nor log segments found. Creating new log.
I20260929 10:58:29.456121 28664 log.cc:824] T f5274ab4f7ad40f5838c325084064677 P 7d9d31444489446fad9dcfe06ebcb0d7: Log is configured to *not* fsync() on all Append() calls
I20260929 10:58:29.456459 28666 log.cc:824] T f5274ab4f7ad40f5838c325084064677 P cbee193e29874985890bcf6bc4bcb668: Log is configured to *not* fsync() on all Append() calls
I20260929 10:58:29.456459 28665 log.cc:824] T f5274ab4f7ad40f5838c325084064677 P 90773ff7f66d457498922c8b38714fa4: Log is configured to *not* fsync() on all Append() calls
I20260929 10:58:29.457288 28666 tablet_bootstrap.cc:492] T f5274ab4f7ad40f5838c325084064677 P cbee193e29874985890bcf6bc4bcb668: No bootstrap required, opened a new log
I20260929 10:58:29.457296 28664 tablet_bootstrap.cc:492] T f5274ab4f7ad40f5838c325084064677 P 7d9d31444489446fad9dcfe06ebcb0d7: No bootstrap required, opened a new log
I20260929 10:58:29.457368 28666 ts_tablet_manager.cc:1397] T f5274ab4f7ad40f5838c325084064677 P cbee193e29874985890bcf6bc4bcb668: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 10:58:29.457368 28664 ts_tablet_manager.cc:1397] T f5274ab4f7ad40f5838c325084064677 P 7d9d31444489446fad9dcfe06ebcb0d7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 10:58:29.457486 28665 tablet_bootstrap.cc:492] T f5274ab4f7ad40f5838c325084064677 P 90773ff7f66d457498922c8b38714fa4: No bootstrap required, opened a new log
I20260929 10:58:29.457578 28665 ts_tablet_manager.cc:1397] T f5274ab4f7ad40f5838c325084064677 P 90773ff7f66d457498922c8b38714fa4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 10:58:29.459129 28665 raft_consensus.cc:348] T f5274ab4f7ad40f5838c325084064677 P 90773ff7f66d457498922c8b38714fa4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9d31444489446fad9dcfe06ebcb0d7" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 36095 } } peers { permanent_uuid: "cbee193e29874985890bcf6bc4bcb668" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 36469 } } peers { permanent_uuid: "90773ff7f66d457498922c8b38714fa4" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 37233 } }
I20260929 10:58:29.459312 28665 raft_consensus.cc:374] T f5274ab4f7ad40f5838c325084064677 P 90773ff7f66d457498922c8b38714fa4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:58:29.459359 28665 raft_consensus.cc:729] T f5274ab4f7ad40f5838c325084064677 P 90773ff7f66d457498922c8b38714fa4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90773ff7f66d457498922c8b38714fa4, State: Initialized, Role: FOLLOWER
I20260929 10:58:29.459499 28665 consensus_queue.cc:260] T f5274ab4f7ad40f5838c325084064677 P 90773ff7f66d457498922c8b38714fa4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9d31444489446fad9dcfe06ebcb0d7" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 36095 } } peers { permanent_uuid: "cbee193e29874985890bcf6bc4bcb668" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 36469 } } peers { permanent_uuid: "90773ff7f66d457498922c8b38714fa4" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 37233 } }
I20260929 10:58:29.459615 28666 raft_consensus.cc:348] T f5274ab4f7ad40f5838c325084064677 P cbee193e29874985890bcf6bc4bcb668 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9d31444489446fad9dcfe06ebcb0d7" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 36095 } } peers { permanent_uuid: "cbee193e29874985890bcf6bc4bcb668" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 36469 } } peers { permanent_uuid: "90773ff7f66d457498922c8b38714fa4" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 37233 } }
I20260929 10:58:29.459671 28664 raft_consensus.cc:348] T f5274ab4f7ad40f5838c325084064677 P 7d9d31444489446fad9dcfe06ebcb0d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9d31444489446fad9dcfe06ebcb0d7" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 36095 } } peers { permanent_uuid: "cbee193e29874985890bcf6bc4bcb668" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 36469 } } peers { permanent_uuid: "90773ff7f66d457498922c8b38714fa4" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 37233 } }
I20260929 10:58:29.459807 28664 raft_consensus.cc:374] T f5274ab4f7ad40f5838c325084064677 P 7d9d31444489446fad9dcfe06ebcb0d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:58:29.459807 28666 raft_consensus.cc:374] T f5274ab4f7ad40f5838c325084064677 P cbee193e29874985890bcf6bc4bcb668 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:58:29.459812 28665 ts_tablet_manager.cc:1428] T f5274ab4f7ad40f5838c325084064677 P 90773ff7f66d457498922c8b38714fa4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 10:58:29.459843 28666 raft_consensus.cc:729] T f5274ab4f7ad40f5838c325084064677 P cbee193e29874985890bcf6bc4bcb668 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbee193e29874985890bcf6bc4bcb668, State: Initialized, Role: FOLLOWER
I20260929 10:58:29.459843 28664 raft_consensus.cc:729] T f5274ab4f7ad40f5838c325084064677 P 7d9d31444489446fad9dcfe06ebcb0d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d9d31444489446fad9dcfe06ebcb0d7, State: Initialized, Role: FOLLOWER
I20260929 10:58:29.459812 28650 heartbeater.cc:503] Master 127.10.238.254:34325 was elected leader, sending a full tablet report...
I20260929 10:58:29.459969 28664 consensus_queue.cc:260] T f5274ab4f7ad40f5838c325084064677 P 7d9d31444489446fad9dcfe06ebcb0d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9d31444489446fad9dcfe06ebcb0d7" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 36095 } } peers { permanent_uuid: "cbee193e29874985890bcf6bc4bcb668" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 36469 } } peers { permanent_uuid: "90773ff7f66d457498922c8b38714fa4" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 37233 } }
I20260929 10:58:29.459969 28666 consensus_queue.cc:260] T f5274ab4f7ad40f5838c325084064677 P cbee193e29874985890bcf6bc4bcb668 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9d31444489446fad9dcfe06ebcb0d7" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 36095 } } peers { permanent_uuid: "cbee193e29874985890bcf6bc4bcb668" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 36469 } } peers { permanent_uuid: "90773ff7f66d457498922c8b38714fa4" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 37233 } }
I20260929 10:58:29.460263 28666 ts_tablet_manager.cc:1428] T f5274ab4f7ad40f5838c325084064677 P cbee193e29874985890bcf6bc4bcb668: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 10:58:29.460291 28664 ts_tablet_manager.cc:1428] T f5274ab4f7ad40f5838c325084064677 P 7d9d31444489446fad9dcfe06ebcb0d7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 10:58:29.460412 28382 heartbeater.cc:503] Master 127.10.238.254:34325 was elected leader, sending a full tablet report...
I20260929 10:58:29.460413 28516 heartbeater.cc:503] Master 127.10.238.254:34325 was elected leader, sending a full tablet report...
W20260929 10:58:29.507614 28517 tablet.cc:2369] T f5274ab4f7ad40f5838c325084064677 P cbee193e29874985890bcf6bc4bcb668: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:58:29.510880 28672 raft_consensus.cc:482] T f5274ab4f7ad40f5838c325084064677 P cbee193e29874985890bcf6bc4bcb668 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:58:29.510985 28672 raft_consensus.cc:504] T f5274ab4f7ad40f5838c325084064677 P cbee193e29874985890bcf6bc4bcb668 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9d31444489446fad9dcfe06ebcb0d7" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 36095 } } peers { permanent_uuid: "cbee193e29874985890bcf6bc4bcb668" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 36469 } } peers { permanent_uuid: "90773ff7f66d457498922c8b38714fa4" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 37233 } }
I20260929 10:58:29.511497 28672 leader_election.cc:288] T f5274ab4f7ad40f5838c325084064677 P cbee193e29874985890bcf6bc4bcb668 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d9d31444489446fad9dcfe06ebcb0d7 (127.10.238.193:36095), 90773ff7f66d457498922c8b38714fa4 (127.10.238.195:37233)
I20260929 10:58:29.514546 28518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:29.511754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.195:37233 (local address 127.10.238.194:47531)
0929 10:58:29.511825 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:29.511842 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 10:58:29.511903 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:58:29.512084 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:58:29.512085 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:58:29.512086 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:58:29.512166 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:29.512167 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:29.512851 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:29.512852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:29.513693 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:29.513696 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:29.514280 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:29.514281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:29.514361 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:29.514364 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 10:58:29.514427 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 10:58:29.514471 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":32}
I20260929 10:58:29.514546 28663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:29.511915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:47531 (local address 127.10.238.195:37233)
0929 10:58:29.511959 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:29.511961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:29.511969 (+ 8us) server_negotiation.cc:407] Connection header received
0929 10:58:29.511997 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:29.511997 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:29.512006 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:29.512079 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:58:29.512274 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:29.512765 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:29.513794 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:29.514190 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:29.514280 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:29.514289 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:29.514446 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 10:58:29.514486 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 10:58:29.514735 28662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:29.511884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:57565 (local address 127.10.238.193:36095)
0929 10:58:29.511937 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:29.511939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:29.512048 (+ 109us) server_negotiation.cc:407] Connection header received
0929 10:58:29.512149 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:29.512150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:29.512161 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:29.512229 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:58:29.512402 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:29.512871 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:29.513912 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:29.514348 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:29.514456 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:29.514465 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:29.514637 (+ 172us) server_negotiation.cc:299] Negotiation successful
0929 10:58:29.514676 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260929 10:58:29.514739 28673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:29.511756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:36095 (local address 127.10.238.194:57565)
0929 10:58:29.511960 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:29.511970 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 10:58:29.512059 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:58:29.512246 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:58:29.512247 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:58:29.512248 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:58:29.512327 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:29.512329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:29.512961 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:29.512962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:29.513808 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:29.513811 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:29.514456 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:29.514457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:29.514539 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:29.514543 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 10:58:29.514617 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 10:58:29.514658 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":17,"thread_start_us":110,"threads_started":1}
I20260929 10:58:29.514909 28604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5274ab4f7ad40f5838c325084064677" candidate_uuid: "cbee193e29874985890bcf6bc4bcb668" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90773ff7f66d457498922c8b38714fa4" is_pre_election: true
I20260929 10:58:29.515045 28336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5274ab4f7ad40f5838c325084064677" candidate_uuid: "cbee193e29874985890bcf6bc4bcb668" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d9d31444489446fad9dcfe06ebcb0d7" is_pre_election: true
I20260929 10:58:29.515105 28604 raft_consensus.cc:2454] T f5274ab4f7ad40f5838c325084064677 P 90773ff7f66d457498922c8b38714fa4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cbee193e29874985890bcf6bc4bcb668 in term 0.
I20260929 10:58:29.515177 28336 raft_consensus.cc:2454] T f5274ab4f7ad40f5838c325084064677 P 7d9d31444489446fad9dcfe06ebcb0d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cbee193e29874985890bcf6bc4bcb668 in term 0.
I20260929 10:58:29.515395 28406 leader_election.cc:302] T f5274ab4f7ad40f5838c325084064677 P cbee193e29874985890bcf6bc4bcb668 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d9d31444489446fad9dcfe06ebcb0d7, cbee193e29874985890bcf6bc4bcb668; no voters:
I20260929 10:58:29.515520 28672 raft_consensus.cc:2781] T f5274ab4f7ad40f5838c325084064677 P cbee193e29874985890bcf6bc4bcb668 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:58:29.515576 28672 raft_consensus.cc:482] T f5274ab4f7ad40f5838c325084064677 P cbee193e29874985890bcf6bc4bcb668 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:58:29.515599 28672 raft_consensus.cc:3037] T f5274ab4f7ad40f5838c325084064677 P cbee193e29874985890bcf6bc4bcb668 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:58:29.516389 28672 raft_consensus.cc:504] T f5274ab4f7ad40f5838c325084064677 P cbee193e29874985890bcf6bc4bcb668 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9d31444489446fad9dcfe06ebcb0d7" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 36095 } } peers { permanent_uuid: "cbee193e29874985890bcf6bc4bcb668" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 36469 } } peers { permanent_uuid: "90773ff7f66d457498922c8b38714fa4" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 37233 } }
I20260929 10:58:29.516599 28672 leader_election.cc:288] T f5274ab4f7ad40f5838c325084064677 P cbee193e29874985890bcf6bc4bcb668 [CANDIDATE]: Term 1 election: Requested vote from peers 7d9d31444489446fad9dcfe06ebcb0d7 (127.10.238.193:36095), 90773ff7f66d457498922c8b38714fa4 (127.10.238.195:37233)
I20260929 10:58:29.516702 28336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5274ab4f7ad40f5838c325084064677" candidate_uuid: "cbee193e29874985890bcf6bc4bcb668" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d9d31444489446fad9dcfe06ebcb0d7"
I20260929 10:58:29.516721 28604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5274ab4f7ad40f5838c325084064677" candidate_uuid: "cbee193e29874985890bcf6bc4bcb668" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90773ff7f66d457498922c8b38714fa4"
I20260929 10:58:29.516780 28336 raft_consensus.cc:3037] T f5274ab4f7ad40f5838c325084064677 P 7d9d31444489446fad9dcfe06ebcb0d7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:58:29.516794 28604 raft_consensus.cc:3037] T f5274ab4f7ad40f5838c325084064677 P 90773ff7f66d457498922c8b38714fa4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:58:29.517637 28604 raft_consensus.cc:2454] T f5274ab4f7ad40f5838c325084064677 P 90773ff7f66d457498922c8b38714fa4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbee193e29874985890bcf6bc4bcb668 in term 1.
I20260929 10:58:29.517783 28336 raft_consensus.cc:2454] T f5274ab4f7ad40f5838c325084064677 P 7d9d31444489446fad9dcfe06ebcb0d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbee193e29874985890bcf6bc4bcb668 in term 1.
I20260929 10:58:29.517839 28404 leader_election.cc:302] T f5274ab4f7ad40f5838c325084064677 P cbee193e29874985890bcf6bc4bcb668 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90773ff7f66d457498922c8b38714fa4, cbee193e29874985890bcf6bc4bcb668; no voters:
I20260929 10:58:29.517925 28672 raft_consensus.cc:2781] T f5274ab4f7ad40f5838c325084064677 P cbee193e29874985890bcf6bc4bcb668 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:58:29.518090 28672 raft_consensus.cc:686] T f5274ab4f7ad40f5838c325084064677 P cbee193e29874985890bcf6bc4bcb668 [term 1 LEADER]: Becoming Leader. State: Replica: cbee193e29874985890bcf6bc4bcb668, State: Running, Role: LEADER
I20260929 10:58:29.518229 28672 consensus_queue.cc:237] T f5274ab4f7ad40f5838c325084064677 P cbee193e29874985890bcf6bc4bcb668 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9d31444489446fad9dcfe06ebcb0d7" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 36095 } } peers { permanent_uuid: "cbee193e29874985890bcf6bc4bcb668" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 36469 } } peers { permanent_uuid: "90773ff7f66d457498922c8b38714fa4" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 37233 } }
I20260929 10:58:29.519146 28189 catalog_manager.cc:5495] T f5274ab4f7ad40f5838c325084064677 P cbee193e29874985890bcf6bc4bcb668 reported cstate change: term changed from 0 to 1, leader changed from <none> to cbee193e29874985890bcf6bc4bcb668 (127.10.238.194). New cstate: current_term: 1 leader_uuid: "cbee193e29874985890bcf6bc4bcb668" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9d31444489446fad9dcfe06ebcb0d7" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cbee193e29874985890bcf6bc4bcb668" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 36469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90773ff7f66d457498922c8b38714fa4" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 37233 } health_report { overall_health: UNKNOWN } } }
I20260929 10:58:29.541617 28660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:29.539186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53558 (local address 127.10.238.194:36469)
0929 10:58:29.539249 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:29.539251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:29.539259 (+ 8us) server_negotiation.cc:407] Connection header received
0929 10:58:29.539296 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:29.539296 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:29.539312 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:29.539393 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:58:29.539538 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:29.540014 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:29.540832 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:29.541028 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:29.541106 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:29.541279 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:58:29.541380 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:58:29.541481 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:29.541502 (+ 21us) server_negotiation.cc:299] Negotiation successful
0929 10:58:29.541523 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 10:58:29.543588 28604 raft_consensus.cc:1261] T f5274ab4f7ad40f5838c325084064677 P 90773ff7f66d457498922c8b38714fa4 [term 1 FOLLOWER]: Refusing update from remote peer cbee193e29874985890bcf6bc4bcb668: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:58:29.543588 28336 raft_consensus.cc:1261] T f5274ab4f7ad40f5838c325084064677 P 7d9d31444489446fad9dcfe06ebcb0d7 [term 1 FOLLOWER]: Refusing update from remote peer cbee193e29874985890bcf6bc4bcb668: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:58:29.543972 28674 consensus_queue.cc:1035] T f5274ab4f7ad40f5838c325084064677 P cbee193e29874985890bcf6bc4bcb668 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90773ff7f66d457498922c8b38714fa4" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 37233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:58:29.544111 28672 consensus_queue.cc:1035] T f5274ab4f7ad40f5838c325084064677 P cbee193e29874985890bcf6bc4bcb668 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d9d31444489446fad9dcfe06ebcb0d7" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:58:29.547600 28679 mvcc.cc:205] Tried to move back new op lower bound from 7334623271087280128 to 7334623270987427840. Current Snapshot: MvccSnapshot[applied={T|T < 7334623271087280128 or (T in {7334623271087280128})}]
I20260929 10:58:29.549247 28191 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:58:29.549396 28191 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:58:29.550196 28191 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f481b8f389ae42728c77739eb8d2b543: Sending DeleteTablet for 3 replicas of tablet f5274ab4f7ad40f5838c325084064677
I20260929 10:58:29.550560 28316 tablet_service.cc:1514] Processing DeleteTablet for tablet f5274ab4f7ad40f5838c325084064677 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:58:29 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:54170
I20260929 10:58:29.550585 28450 tablet_service.cc:1514] Processing DeleteTablet for tablet f5274ab4f7ad40f5838c325084064677 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:58:29 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:53552
I20260929 10:58:29.550802 28686 tablet_replica.cc:330] T f5274ab4f7ad40f5838c325084064677 P 7d9d31444489446fad9dcfe06ebcb0d7: stopping tablet replica
I20260929 10:58:29.550802 28584 tablet_service.cc:1514] Processing DeleteTablet for tablet f5274ab4f7ad40f5838c325084064677 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:58:29 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:50150
I20260929 10:58:29.550922 28686 raft_consensus.cc:2229] T f5274ab4f7ad40f5838c325084064677 P 7d9d31444489446fad9dcfe06ebcb0d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:58:29.551039 28686 raft_consensus.cc:2258] T f5274ab4f7ad40f5838c325084064677 P 7d9d31444489446fad9dcfe06ebcb0d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:58:29.551122 28688 tablet_replica.cc:330] T f5274ab4f7ad40f5838c325084064677 P 90773ff7f66d457498922c8b38714fa4: stopping tablet replica
I20260929 10:58:29.551209 28688 raft_consensus.cc:2229] T f5274ab4f7ad40f5838c325084064677 P 90773ff7f66d457498922c8b38714fa4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:58:29.551218 28687 tablet_replica.cc:330] T f5274ab4f7ad40f5838c325084064677 P cbee193e29874985890bcf6bc4bcb668: stopping tablet replica
I20260929 10:58:29.551299 28688 raft_consensus.cc:2258] T f5274ab4f7ad40f5838c325084064677 P 90773ff7f66d457498922c8b38714fa4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:58:29.551319 28687 raft_consensus.cc:2229] T f5274ab4f7ad40f5838c325084064677 P cbee193e29874985890bcf6bc4bcb668 [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:58:29.551385 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 28253
I20260929 10:58:29.551412 28687 raft_consensus.cc:2258] T f5274ab4f7ad40f5838c325084064677 P cbee193e29874985890bcf6bc4bcb668 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:58:29.552333 28688 ts_tablet_manager.cc:1905] T f5274ab4f7ad40f5838c325084064677 P 90773ff7f66d457498922c8b38714fa4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:58:29.552362 28687 ts_tablet_manager.cc:1905] T f5274ab4f7ad40f5838c325084064677 P cbee193e29874985890bcf6bc4bcb668: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:58:29.554541 28688 ts_tablet_manager.cc:1918] T f5274ab4f7ad40f5838c325084064677 P 90773ff7f66d457498922c8b38714fa4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:58:29.554539 28687 ts_tablet_manager.cc:1918] T f5274ab4f7ad40f5838c325084064677 P cbee193e29874985890bcf6bc4bcb668: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:58:29.554620 28688 log.cc:1196] T f5274ab4f7ad40f5838c325084064677 P 90773ff7f66d457498922c8b38714fa4: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/ts-2/wal/wals/f5274ab4f7ad40f5838c325084064677
I20260929 10:58:29.554618 28687 log.cc:1196] T f5274ab4f7ad40f5838c325084064677 P cbee193e29874985890bcf6bc4bcb668: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790679274510869-11195-0/minicluster-data/ts-1/wal/wals/f5274ab4f7ad40f5838c325084064677
I20260929 10:58:29.554874 28688 ts_tablet_manager.cc:1939] T f5274ab4f7ad40f5838c325084064677 P 90773ff7f66d457498922c8b38714fa4: Deleting consensus metadata
I20260929 10:58:29.554912 28687 ts_tablet_manager.cc:1939] T f5274ab4f7ad40f5838c325084064677 P cbee193e29874985890bcf6bc4bcb668: Deleting consensus metadata
I20260929 10:58:29.555331 28176 catalog_manager.cc:4841] TS cbee193e29874985890bcf6bc4bcb668 (127.10.238.194:36469): tablet f5274ab4f7ad40f5838c325084064677 (table test-table [id=a8554c4a667f475b8cbb0570355da174]) successfully deleted
I20260929 10:58:29.555433 28176 catalog_manager.cc:4841] TS 90773ff7f66d457498922c8b38714fa4 (127.10.238.195:37233): tablet f5274ab4f7ad40f5838c325084064677 (table test-table [id=a8554c4a667f475b8cbb0570355da174]) successfully deleted
W20260929 10:58:29.558452 28178 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.238.193:36095 (error 108)
I20260929 10:58:29.558645 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 28387
I20260929 10:58:29.559620 28659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:29.559463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:36095 (local address 127.0.0.1:54180)
0929 10:58:29.559534 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:29.559582 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.193:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 10:58:29.559763 28178 catalog_manager.cc:4584] TS 7d9d31444489446fad9dcfe06ebcb0d7 (127.10.238.193:36095): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f5274ab4f7ad40f5838c325084064677: Network error: Client connection negotiation failed: client connection to 127.10.238.193:36095: connect: Connection refused (error 111)
I20260929 10:58:29.563920 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 28521
I20260929 10:58:29.568759 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 28159
2026-09-29T10:58:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1951 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T10:58:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:58:29Z Disabled control of system clock
I20260929 10:58:29.598045 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:41903
--webserver_interface=127.10.238.254
--webserver_port=0
--builtin_ntp_servers=127.10.238.212:38077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:41903
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 10:58:29.681384 28693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:58:29.681603 28693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:58:29.681624 28693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:58:29.682781 28693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:58:29.682823 28693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:58:29.682837 28693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:58:29.682849 28693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:58:29.682861 28693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:58:29.684114 28693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:38077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:41903
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:41903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:58:29.684309 28693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:58:29.684536 28693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 10:58:29.689618 28693 server_base.cc:1034] running on GCE node
W20260929 10:58:29.689769 28699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:29.689849 28698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:29.689594 28701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:29.690200 28693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:58:29.690538 28693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:58:29.691716 28693 hybrid_clock.cc:648] HybridClock initialized: now 1790679509691691 us; error 39 us; skew 500 ppm
I20260929 10:58:29.693046 28693 webserver.cc:460] Webserver started at http://127.10.238.254:36741/ using document root <none> and password file <none>
I20260929 10:58:29.693363 28693 fs_manager.cc:359] Metadata directory not provided
I20260929 10:58:29.693415 28693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:58:29.693509 28693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:58:29.694517 28693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/master-0/data/instance:
uuid: "962e8d5532774e1fb590d7f41f491c39"
format_stamp: "Formatted at 2026-09-29 10:58:29 on dist-test-slave-8nrt"
server_key: "ccb5607535b2fd47a924af6018cd740c"
server_key_iv: "5de2cfcd3b0dc0eba34b06e87b0bbae2"
server_key_version: "encryptionkey@0"
I20260929 10:58:29.694810 28693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/master-0/wal/instance:
uuid: "962e8d5532774e1fb590d7f41f491c39"
format_stamp: "Formatted at 2026-09-29 10:58:29 on dist-test-slave-8nrt"
server_key: "ccb5607535b2fd47a924af6018cd740c"
server_key_iv: "5de2cfcd3b0dc0eba34b06e87b0bbae2"
server_key_version: "encryptionkey@0"
I20260929 10:58:29.696036 28693 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 10:58:29.696702 28707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:29.696869 28693 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:58:29.696946 28693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "962e8d5532774e1fb590d7f41f491c39"
format_stamp: "Formatted at 2026-09-29 10:58:29 on dist-test-slave-8nrt"
server_key: "ccb5607535b2fd47a924af6018cd740c"
server_key_iv: "5de2cfcd3b0dc0eba34b06e87b0bbae2"
server_key_version: "encryptionkey@0"
I20260929 10:58:29.696997 28693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:58:29.725404 28693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:58:29.725710 28693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:58:29.725821 28693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:58:29.729708 28693 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.254:41903
I20260929 10:58:29.729749 28759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.254:41903 every 8 connection(s)
I20260929 10:58:29.730108 28693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:58:29.731130 28760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:58:29.732508 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 28693
I20260929 10:58:29.732625 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/master-0/wal/instance
I20260929 10:58:29.732821 11195 external_mini_cluster.cc:1428] Setting key e69f4a5f1f98d76d830e854a32e75e26
I20260929 10:58:29.733935 28760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 962e8d5532774e1fb590d7f41f491c39: Bootstrap starting.
I20260929 10:58:29.734665 28760 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 962e8d5532774e1fb590d7f41f491c39: Neither blocks nor log segments found. Creating new log.
I20260929 10:58:29.734998 28760 log.cc:824] T 00000000000000000000000000000000 P 962e8d5532774e1fb590d7f41f491c39: Log is configured to *not* fsync() on all Append() calls
I20260929 10:58:29.735737 28760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 962e8d5532774e1fb590d7f41f491c39: No bootstrap required, opened a new log
I20260929 10:58:29.737970 28760 raft_consensus.cc:348] T 00000000000000000000000000000000 P 962e8d5532774e1fb590d7f41f491c39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "962e8d5532774e1fb590d7f41f491c39" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 41903 } }
I20260929 10:58:29.738139 28760 raft_consensus.cc:374] T 00000000000000000000000000000000 P 962e8d5532774e1fb590d7f41f491c39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:58:29.738194 28760 raft_consensus.cc:729] T 00000000000000000000000000000000 P 962e8d5532774e1fb590d7f41f491c39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 962e8d5532774e1fb590d7f41f491c39, State: Initialized, Role: FOLLOWER
I20260929 10:58:29.738334 28760 consensus_queue.cc:260] T 00000000000000000000000000000000 P 962e8d5532774e1fb590d7f41f491c39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "962e8d5532774e1fb590d7f41f491c39" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 41903 } }
I20260929 10:58:29.738416 28760 raft_consensus.cc:388] T 00000000000000000000000000000000 P 962e8d5532774e1fb590d7f41f491c39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:58:29.738454 28760 raft_consensus.cc:482] T 00000000000000000000000000000000 P 962e8d5532774e1fb590d7f41f491c39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:58:29.738505 28760 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 962e8d5532774e1fb590d7f41f491c39 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:58:29.739053 28763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:29.733390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34262 (local address 127.10.238.254:41903)
0929 10:58:29.733613 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:29.733622 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:29.733657 (+ 35us) server_negotiation.cc:407] Connection header received
0929 10:58:29.734204 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:29.734213 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:29.734311 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:29.734477 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:29.734632 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:29.735954 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:29.736761 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:29.737415 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:29.737568 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:29.738287 (+ 719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:29.738292 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:29.738292 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:58:29.738507 (+ 215us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:29.738554 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:58:29.738569 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:29.738632 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:29.738758 (+ 126us) server_negotiation.cc:299] Negotiation successful
0929 10:58:29.738849 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":84,"threads_started":1}
I20260929 10:58:29.739264 28760 raft_consensus.cc:504] T 00000000000000000000000000000000 P 962e8d5532774e1fb590d7f41f491c39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "962e8d5532774e1fb590d7f41f491c39" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 41903 } }
I20260929 10:58:29.739372 28760 leader_election.cc:302] T 00000000000000000000000000000000 P 962e8d5532774e1fb590d7f41f491c39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 962e8d5532774e1fb590d7f41f491c39; no voters:
I20260929 10:58:29.739532 28760 leader_election.cc:288] T 00000000000000000000000000000000 P 962e8d5532774e1fb590d7f41f491c39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:58:29.739570 28765 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 962e8d5532774e1fb590d7f41f491c39 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:58:29.739709 28765 raft_consensus.cc:686] T 00000000000000000000000000000000 P 962e8d5532774e1fb590d7f41f491c39 [term 1 LEADER]: Becoming Leader. State: Replica: 962e8d5532774e1fb590d7f41f491c39, State: Running, Role: LEADER
I20260929 10:58:29.739774 28760 sys_catalog.cc:564] T 00000000000000000000000000000000 P 962e8d5532774e1fb590d7f41f491c39 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:58:29.739897 28765 consensus_queue.cc:237] T 00000000000000000000000000000000 P 962e8d5532774e1fb590d7f41f491c39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "962e8d5532774e1fb590d7f41f491c39" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 41903 } }
I20260929 10:58:29.740329 28766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 962e8d5532774e1fb590d7f41f491c39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "962e8d5532774e1fb590d7f41f491c39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "962e8d5532774e1fb590d7f41f491c39" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 41903 } } }
I20260929 10:58:29.740447 28766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 962e8d5532774e1fb590d7f41f491c39 [sys.catalog]: This master's current role is: LEADER
I20260929 10:58:29.740682 28774 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:58:29.741163 28774 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:58:29.741113 28767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 962e8d5532774e1fb590d7f41f491c39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 962e8d5532774e1fb590d7f41f491c39. Latest consensus state: current_term: 1 leader_uuid: "962e8d5532774e1fb590d7f41f491c39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "962e8d5532774e1fb590d7f41f491c39" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 41903 } } }
I20260929 10:58:29.741389 28767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 962e8d5532774e1fb590d7f41f491c39 [sys.catalog]: This master's current role is: LEADER
I20260929 10:58:29.742906 28774 catalog_manager.cc:1347] Generated new cluster ID: 32b28959cc3e4800a87d6a925922628a
I20260929 10:58:29.742959 28774 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:58:29.763163 28774 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:58:29.763705 28774 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:58:29.766657 28774 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 962e8d5532774e1fb590d7f41f491c39: Generated new TSK 0
I20260929 10:58:29.766816 28774 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:58:29.769306 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.193:0
--local_ip_for_outbound_sockets=127.10.238.193
--webserver_interface=127.10.238.193
--webserver_port=0
--tserver_master_addrs=127.10.238.254:41903
--builtin_ntp_servers=127.10.238.212:38077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 10:58:29.857298 28784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:58:29.857554 28784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:58:29.857594 28784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:58:29.858757 28784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:58:29.858824 28784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:58:29.858866 28784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.193
I20260929 10:58:29.860183 28784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:38077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.238.193
--webserver_port=0
--tserver_master_addrs=127.10.238.254:41903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.193
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:58:29.860451 28784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:58:29.860759 28784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:58:29.866863 28792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:29.866874 28789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:29.866874 28790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:29.867259 28784 server_base.cc:1034] running on GCE node
I20260929 10:58:29.867442 28784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:58:29.867712 28784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:58:29.868868 28784 hybrid_clock.cc:648] HybridClock initialized: now 1790679509868847 us; error 37 us; skew 500 ppm
I20260929 10:58:29.870286 28784 webserver.cc:460] Webserver started at http://127.10.238.193:46655/ using document root <none> and password file <none>
I20260929 10:58:29.870555 28784 fs_manager.cc:359] Metadata directory not provided
I20260929 10:58:29.870600 28784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:58:29.870765 28784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:58:29.871843 28784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/ts-0/data/instance:
uuid: "82406c217cae4759ae4a406064a5cdfc"
format_stamp: "Formatted at 2026-09-29 10:58:29 on dist-test-slave-8nrt"
server_key: "fefd3899f705a6278acc50b43fd1b213"
server_key_iv: "de01f6dcab59905bf021c19a87aeba2b"
server_key_version: "encryptionkey@0"
I20260929 10:58:29.872206 28784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/ts-0/wal/instance:
uuid: "82406c217cae4759ae4a406064a5cdfc"
format_stamp: "Formatted at 2026-09-29 10:58:29 on dist-test-slave-8nrt"
server_key: "fefd3899f705a6278acc50b43fd1b213"
server_key_iv: "de01f6dcab59905bf021c19a87aeba2b"
server_key_version: "encryptionkey@0"
I20260929 10:58:29.873555 28784 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 10:58:29.874356 28798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:29.874573 28784 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:58:29.874657 28784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/ts-0/wal
uuid: "82406c217cae4759ae4a406064a5cdfc"
format_stamp: "Formatted at 2026-09-29 10:58:29 on dist-test-slave-8nrt"
server_key: "fefd3899f705a6278acc50b43fd1b213"
server_key_iv: "de01f6dcab59905bf021c19a87aeba2b"
server_key_version: "encryptionkey@0"
I20260929 10:58:29.874711 28784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:58:29.895294 28784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:58:29.895639 28784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:58:29.895792 28784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:58:29.896024 28784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:58:29.896386 28784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:58:29.896445 28784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:29.896488 28784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:58:29.896514 28784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:29.902323 28784 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.193:45531
I20260929 10:58:29.902431 28911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.193:45531 every 8 connection(s)
I20260929 10:58:29.902765 28784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
I20260929 10:58:29.906731 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 28784
I20260929 10:58:29.906847 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/ts-0/wal/instance
I20260929 10:58:29.907044 11195 external_mini_cluster.cc:1428] Setting key d4d712b3dd2f8c0da0e67a9e15fb9839
I20260929 10:58:29.907965 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.194:0
--local_ip_for_outbound_sockets=127.10.238.194
--webserver_interface=127.10.238.194
--webserver_port=0
--tserver_master_addrs=127.10.238.254:41903
--builtin_ntp_servers=127.10.238.212:38077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 10:58:29.908443 28763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:29.903487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:36105 (local address 127.10.238.254:41903)
0929 10:58:29.903531 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:29.903533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:29.903792 (+ 259us) server_negotiation.cc:407] Connection header received
0929 10:58:29.904255 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:29.904256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:29.904271 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:29.904325 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:29.905042 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:29.905609 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:29.906773 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:29.907037 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:29.907088 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:29.908178 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:29.908179 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:29.908179 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:58:29.908184 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:29.908201 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:58:29.908206 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:29.908257 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:29.908330 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 10:58:29.908357 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 10:58:29.908672 28914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:29.903485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:41903 (local address 127.10.238.193:36105)
0929 10:58:29.903669 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:29.903702 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 10:58:29.904175 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:58:29.904379 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:58:29.904382 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:58:29.904390 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:58:29.904969 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:29.904974 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:29.905686 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:29.905687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:29.906682 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:29.906687 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:29.907148 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:29.907150 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:29.907342 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:29.907965 (+ 623us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:58:29.907966 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:58:29.907995 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:58:29.908018 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:58:29.908258 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:58:29.908265 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:58:29.908268 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 10:58:29.908336 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 10:58:29.908439 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":96,"threads_started":1}
I20260929 10:58:29.909124 28912 heartbeater.cc:347] Connected to a master server at 127.10.238.254:41903
I20260929 10:58:29.909217 28912 heartbeater.cc:464] Registering TS with master...
I20260929 10:58:29.909443 28912 heartbeater.cc:511] Master 127.10.238.254:41903 requested a full tablet report, sending...
I20260929 10:58:29.909981 28724 ts_manager.cc:194] Registered new tserver with Master: 82406c217cae4759ae4a406064a5cdfc (127.10.238.193:45531)
I20260929 10:58:29.910600 28724 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.238.193:36105
W20260929 10:58:29.991777 28915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:58:29.991976 28915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:58:29.991997 28915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:58:29.993157 28915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:58:29.993197 28915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:58:29.993223 28915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.194
I20260929 10:58:29.994660 28915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:38077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.238.194
--webserver_port=0
--tserver_master_addrs=127.10.238.254:41903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28915
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.194
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:58:29.994896 28915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:58:29.995194 28915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:58:30.000430 28923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:30.000520 28921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:30.000527 28915 server_base.cc:1034] running on GCE node
W20260929 10:58:30.000464 28920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:30.000852 28915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:58:30.001166 28915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:58:30.002343 28915 hybrid_clock.cc:648] HybridClock initialized: now 1790679510002329 us; error 31 us; skew 500 ppm
I20260929 10:58:30.003563 28915 webserver.cc:460] Webserver started at http://127.10.238.194:34921/ using document root <none> and password file <none>
I20260929 10:58:30.003760 28915 fs_manager.cc:359] Metadata directory not provided
I20260929 10:58:30.003803 28915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:58:30.003890 28915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:58:30.004915 28915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/ts-1/data/instance:
uuid: "7d809a95dd9f49448eb88d18dcd9b083"
format_stamp: "Formatted at 2026-09-29 10:58:29 on dist-test-slave-8nrt"
server_key: "d8c3740d7d176b86f8d6c044bf8905bb"
server_key_iv: "728d38953a07d3659bcafb9b103ee264"
server_key_version: "encryptionkey@0"
I20260929 10:58:30.005216 28915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/ts-1/wal/instance:
uuid: "7d809a95dd9f49448eb88d18dcd9b083"
format_stamp: "Formatted at 2026-09-29 10:58:29 on dist-test-slave-8nrt"
server_key: "d8c3740d7d176b86f8d6c044bf8905bb"
server_key_iv: "728d38953a07d3659bcafb9b103ee264"
server_key_version: "encryptionkey@0"
I20260929 10:58:30.006569 28915 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:58:30.007304 28929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:30.007485 28915 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:58:30.007572 28915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/ts-1/wal
uuid: "7d809a95dd9f49448eb88d18dcd9b083"
format_stamp: "Formatted at 2026-09-29 10:58:29 on dist-test-slave-8nrt"
server_key: "d8c3740d7d176b86f8d6c044bf8905bb"
server_key_iv: "728d38953a07d3659bcafb9b103ee264"
server_key_version: "encryptionkey@0"
I20260929 10:58:30.007627 28915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:58:30.025355 28915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:58:30.025631 28915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:58:30.025731 28915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:58:30.025921 28915 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:58:30.026264 28915 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:58:30.026299 28915 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:30.026321 28915 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:58:30.026376 28915 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:30.032569 28915 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.194:38689
I20260929 10:58:30.032621 29042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.194:38689 every 8 connection(s)
I20260929 10:58:30.032951 28915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
I20260929 10:58:30.033795 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 28915
I20260929 10:58:30.033919 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/ts-1/wal/instance
I20260929 10:58:30.034128 11195 external_mini_cluster.cc:1428] Setting key f2e95e27573d41acd2fcea6e95a32f91
I20260929 10:58:30.035095 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.195:0
--local_ip_for_outbound_sockets=127.10.238.195
--webserver_interface=127.10.238.195
--webserver_port=0
--tserver_master_addrs=127.10.238.254:41903
--builtin_ntp_servers=127.10.238.212:38077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 10:58:30.038664 28763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:30.033888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:33335 (local address 127.10.238.254:41903)
0929 10:58:30.033951 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:30.033954 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:30.034220 (+ 266us) server_negotiation.cc:407] Connection header received
0929 10:58:30.034775 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:30.034776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:30.034788 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:30.034836 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:30.035602 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:30.036103 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:30.037148 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:30.037366 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:30.037449 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:30.038348 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:30.038350 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:30.038350 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:58:30.038355 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:30.038376 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:58:30.038381 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:30.038443 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:30.038556 (+ 113us) server_negotiation.cc:299] Negotiation successful
0929 10:58:30.038584 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 10:58:30.038923 29045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:30.033786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:41903 (local address 127.10.238.194:33335)
0929 10:58:30.034081 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:30.034117 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 10:58:30.034649 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:58:30.034912 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:58:30.034916 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:58:30.034924 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:58:30.035506 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:30.035511 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:30.036230 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:30.036232 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:30.037041 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:30.037043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:30.037456 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:30.037457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:30.037588 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:30.038170 (+ 582us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:58:30.038171 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:58:30.038201 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:58:30.038223 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:58:30.038462 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:58:30.038468 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:58:30.038470 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 10:58:30.038558 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 10:58:30.038706 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":96,"threads_started":1}
I20260929 10:58:30.039445 29043 heartbeater.cc:347] Connected to a master server at 127.10.238.254:41903
I20260929 10:58:30.039536 29043 heartbeater.cc:464] Registering TS with master...
I20260929 10:58:30.039747 29043 heartbeater.cc:511] Master 127.10.238.254:41903 requested a full tablet report, sending...
I20260929 10:58:30.040174 28724 ts_manager.cc:194] Registered new tserver with Master: 7d809a95dd9f49448eb88d18dcd9b083 (127.10.238.194:38689)
I20260929 10:58:30.040647 28724 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.238.194:33335
W20260929 10:58:30.125734 29046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:58:30.125962 29046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:58:30.125985 29046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:58:30.127266 29046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:58:30.127310 29046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:58:30.127337 29046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.195
I20260929 10:58:30.128829 29046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:38077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.238.195
--webserver_port=0
--tserver_master_addrs=127.10.238.254:41903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.195
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:58:30.129096 29046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:58:30.129444 29046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:58:30.134749 29051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:30.134892 29046 server_base.cc:1034] running on GCE node
W20260929 10:58:30.134745 29054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:30.135061 29052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:30.135310 29046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:58:30.135548 29046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:58:30.136713 29046 hybrid_clock.cc:648] HybridClock initialized: now 1790679510136700 us; error 30 us; skew 500 ppm
I20260929 10:58:30.138125 29046 webserver.cc:460] Webserver started at http://127.10.238.195:35849/ using document root <none> and password file <none>
I20260929 10:58:30.138376 29046 fs_manager.cc:359] Metadata directory not provided
I20260929 10:58:30.138424 29046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:58:30.138595 29046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:58:30.139775 29046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/ts-2/data/instance:
uuid: "117d5c2f94f74d839d670b466641d5bd"
format_stamp: "Formatted at 2026-09-29 10:58:30 on dist-test-slave-8nrt"
server_key: "4169a15d533a1efe8dc0c9139bd1d64a"
server_key_iv: "8566376057f61ad1f18f4496ca92083a"
server_key_version: "encryptionkey@0"
I20260929 10:58:30.140204 29046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/ts-2/wal/instance:
uuid: "117d5c2f94f74d839d670b466641d5bd"
format_stamp: "Formatted at 2026-09-29 10:58:30 on dist-test-slave-8nrt"
server_key: "4169a15d533a1efe8dc0c9139bd1d64a"
server_key_iv: "8566376057f61ad1f18f4496ca92083a"
server_key_version: "encryptionkey@0"
I20260929 10:58:30.141782 29046 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 10:58:30.142506 29060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:30.142678 29046 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 10:58:30.142776 29046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/ts-2/wal
uuid: "117d5c2f94f74d839d670b466641d5bd"
format_stamp: "Formatted at 2026-09-29 10:58:30 on dist-test-slave-8nrt"
server_key: "4169a15d533a1efe8dc0c9139bd1d64a"
server_key_iv: "8566376057f61ad1f18f4496ca92083a"
server_key_version: "encryptionkey@0"
I20260929 10:58:30.142853 29046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:58:30.167057 29046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:58:30.167418 29046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:58:30.167582 29046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:58:30.167842 29046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:58:30.168208 29046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:58:30.168263 29046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:30.168308 29046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:58:30.168334 29046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:30.174510 29046 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.195:39819
I20260929 10:58:30.174561 29173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.195:39819 every 8 connection(s)
I20260929 10:58:30.174892 29046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
I20260929 10:58:30.180707 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 29046
I20260929 10:58:30.180868 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/ts-2/wal/instance
I20260929 10:58:30.180979 28763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:30.175756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:34773 (local address 127.10.238.254:41903)
0929 10:58:30.175846 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:30.175849 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:30.176146 (+ 297us) server_negotiation.cc:407] Connection header received
0929 10:58:30.176788 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:30.176790 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:30.176804 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:30.176873 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:30.177700 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:30.178265 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:30.179348 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:30.179540 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:30.179605 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:30.180592 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:30.180594 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:30.180594 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:58:30.180600 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:30.180623 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:58:30.180629 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:30.180715 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:30.180831 (+ 116us) server_negotiation.cc:299] Negotiation successful
0929 10:58:30.180869 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 10:58:30.181159 11195 external_mini_cluster.cc:1428] Setting key 6b438b77791034d4a7eae339b1fbfc60
I20260929 10:58:30.181221 29176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:30.175750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:41903 (local address 127.10.238.195:34773)
0929 10:58:30.175965 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:30.176011 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 10:58:30.176609 (+ 598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:58:30.176966 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:58:30.176970 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:58:30.176980 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:58:30.177584 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:30.177589 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:30.178393 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:30.178395 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:30.179246 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:30.179249 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:30.179665 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:30.179666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:30.179823 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:30.180430 (+ 607us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:58:30.180430 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:58:30.180460 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:58:30.180483 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:58:30.180733 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:58:30.180737 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:58:30.180740 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 10:58:30.180795 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 10:58:30.180943 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":106,"threads_started":1}
I20260929 10:58:30.181931 29174 heartbeater.cc:347] Connected to a master server at 127.10.238.254:41903
I20260929 10:58:30.182045 29174 heartbeater.cc:464] Registering TS with master...
I20260929 10:58:30.182250 29174 heartbeater.cc:511] Master 127.10.238.254:41903 requested a full tablet report, sending...
I20260929 10:58:30.182700 28723 ts_manager.cc:194] Registered new tserver with Master: 117d5c2f94f74d839d670b466641d5bd (127.10.238.195:39819)
I20260929 10:58:30.183097 28723 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.238.195:34773
I20260929 10:58:30.193773 11195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:58:30.198482 28763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:30.195416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34268 (local address 127.10.238.254:41903)
0929 10:58:30.195502 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:30.195504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:30.195592 (+ 88us) server_negotiation.cc:407] Connection header received
0929 10:58:30.195651 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:30.195652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:30.195663 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:30.195722 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:30.195895 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:30.196458 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:30.197365 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:30.197556 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:30.197617 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:30.198230 (+ 613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:30.198231 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:30.198231 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:58:30.198235 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:30.198249 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:58:30.198253 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:30.198296 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:30.198345 (+ 49us) server_negotiation.cc:299] Negotiation successful
0929 10:58:30.198367 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 10:58:30.200363 28723 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34268:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:58:30.202054 28723 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:58:30.208509 29184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:30.204309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51962 (local address 127.10.238.194:38689)
0929 10:58:30.204747 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:30.204752 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:30.204761 (+ 9us) server_negotiation.cc:407] Connection header received
0929 10:58:30.204802 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:30.204803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:30.204816 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:30.204912 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:58:30.205298 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:30.206426 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:30.207609 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:30.208097 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:30.208199 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:30.208300 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:30.208391 (+ 91us) server_negotiation.cc:299] Negotiation successful
0929 10:58:30.208431 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":108,"threads_started":1}
I20260929 10:58:30.208503 29182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:30.204214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.194:38689 (local address 127.0.0.1:51962)
0929 10:58:30.204564 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:30.204576 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:58:30.204640 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:58:30.204936 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:58:30.204939 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:58:30.204942 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:58:30.205012 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:30.205013 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:30.206515 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:30.206516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:30.207524 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:30.207526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:30.208201 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:30.208202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:30.208277 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:30.208289 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 10:58:30.208363 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 10:58:30.208422 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":52,"threads_started":1}
I20260929 10:58:30.208604 29183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:30.204176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:45531 (local address 127.0.0.1:51820)
0929 10:58:30.204528 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:30.204563 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 10:58:30.204640 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:58:30.205195 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:58:30.205196 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:58:30.205198 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:58:30.205275 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:30.205278 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:30.206145 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:30.206146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:30.207353 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:30.207355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:30.208380 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:30.208381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:30.208442 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:30.208446 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 10:58:30.208496 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 10:58:30.208536 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":57,"threads_started":1}
I20260929 10:58:30.208804 29186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:30.204352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51820 (local address 127.10.238.193:45531)
0929 10:58:30.205050 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:30.205054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:30.205063 (+ 9us) server_negotiation.cc:407] Connection header received
0929 10:58:30.205104 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:30.205106 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:30.205121 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:30.205196 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:58:30.205338 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:30.206052 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:30.207454 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:30.208200 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:30.208265 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:30.208347 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:30.208670 (+ 323us) server_negotiation.cc:299] Negotiation successful
0929 10:58:30.208723 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":109,"threads_started":1}
I20260929 10:58:30.209463 28846 tablet_service.cc:1467] Processing CreateTablet for tablet 3c5443a90f7145a4bd5438b82f445dc2 (DEFAULT_TABLE table=test-table [id=2dfd08392d754908b45617936ddeacee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:58:30.209487 28977 tablet_service.cc:1467] Processing CreateTablet for tablet 3c5443a90f7145a4bd5438b82f445dc2 (DEFAULT_TABLE table=test-table [id=2dfd08392d754908b45617936ddeacee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:58:30.209690 29185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:30.204381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.195:39819 (local address 127.0.0.1:40384)
0929 10:58:30.205347 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:30.205358 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 10:58:30.205408 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:58:30.205654 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:58:30.205655 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:58:30.205657 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:58:30.205709 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:30.205710 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:30.206931 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:30.206933 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:30.207955 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:30.207958 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:30.209460 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:30.209461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:30.209527 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:30.209532 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 10:58:30.209584 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 10:58:30.209624 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":946,"mutex_wait_us":8,"thread_start_us":70,"threads_started":1}
I20260929 10:58:30.209836 28846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c5443a90f7145a4bd5438b82f445dc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:58:30.209851 28977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c5443a90f7145a4bd5438b82f445dc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:58:30.209990 29187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:30.204502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40384 (local address 127.10.238.195:39819)
0929 10:58:30.204714 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:30.204719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:30.205466 (+ 747us) server_negotiation.cc:407] Connection header received
0929 10:58:30.205509 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:30.205512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:30.205526 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:30.205591 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:58:30.205781 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:30.206764 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:30.208054 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:30.209277 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:30.209346 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:30.209424 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:30.209859 (+ 435us) server_negotiation.cc:299] Negotiation successful
0929 10:58:30.209904 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":104,"threads_started":1}
I20260929 10:58:30.210575 29108 tablet_service.cc:1467] Processing CreateTablet for tablet 3c5443a90f7145a4bd5438b82f445dc2 (DEFAULT_TABLE table=test-table [id=2dfd08392d754908b45617936ddeacee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:58:30.210896 29108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c5443a90f7145a4bd5438b82f445dc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:58:30.212684 29188 tablet_bootstrap.cc:492] T 3c5443a90f7145a4bd5438b82f445dc2 P 82406c217cae4759ae4a406064a5cdfc: Bootstrap starting.
I20260929 10:58:30.213454 29188 tablet_bootstrap.cc:654] T 3c5443a90f7145a4bd5438b82f445dc2 P 82406c217cae4759ae4a406064a5cdfc: Neither blocks nor log segments found. Creating new log.
I20260929 10:58:30.213554 29189 tablet_bootstrap.cc:492] T 3c5443a90f7145a4bd5438b82f445dc2 P 7d809a95dd9f49448eb88d18dcd9b083: Bootstrap starting.
I20260929 10:58:30.213927 29188 log.cc:824] T 3c5443a90f7145a4bd5438b82f445dc2 P 82406c217cae4759ae4a406064a5cdfc: Log is configured to *not* fsync() on all Append() calls
I20260929 10:58:30.214218 29189 tablet_bootstrap.cc:654] T 3c5443a90f7145a4bd5438b82f445dc2 P 7d809a95dd9f49448eb88d18dcd9b083: Neither blocks nor log segments found. Creating new log.
I20260929 10:58:30.214425 29190 tablet_bootstrap.cc:492] T 3c5443a90f7145a4bd5438b82f445dc2 P 117d5c2f94f74d839d670b466641d5bd: Bootstrap starting.
I20260929 10:58:30.214547 29189 log.cc:824] T 3c5443a90f7145a4bd5438b82f445dc2 P 7d809a95dd9f49448eb88d18dcd9b083: Log is configured to *not* fsync() on all Append() calls
I20260929 10:58:30.214898 29188 tablet_bootstrap.cc:492] T 3c5443a90f7145a4bd5438b82f445dc2 P 82406c217cae4759ae4a406064a5cdfc: No bootstrap required, opened a new log
I20260929 10:58:30.214988 29188 ts_tablet_manager.cc:1397] T 3c5443a90f7145a4bd5438b82f445dc2 P 82406c217cae4759ae4a406064a5cdfc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 10:58:30.215087 29190 tablet_bootstrap.cc:654] T 3c5443a90f7145a4bd5438b82f445dc2 P 117d5c2f94f74d839d670b466641d5bd: Neither blocks nor log segments found. Creating new log.
I20260929 10:58:30.215287 29189 tablet_bootstrap.cc:492] T 3c5443a90f7145a4bd5438b82f445dc2 P 7d809a95dd9f49448eb88d18dcd9b083: No bootstrap required, opened a new log
I20260929 10:58:30.215358 29189 ts_tablet_manager.cc:1397] T 3c5443a90f7145a4bd5438b82f445dc2 P 7d809a95dd9f49448eb88d18dcd9b083: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 10:58:30.215427 29190 log.cc:824] T 3c5443a90f7145a4bd5438b82f445dc2 P 117d5c2f94f74d839d670b466641d5bd: Log is configured to *not* fsync() on all Append() calls
I20260929 10:58:30.216277 29190 tablet_bootstrap.cc:492] T 3c5443a90f7145a4bd5438b82f445dc2 P 117d5c2f94f74d839d670b466641d5bd: No bootstrap required, opened a new log
I20260929 10:58:30.216367 29190 ts_tablet_manager.cc:1397] T 3c5443a90f7145a4bd5438b82f445dc2 P 117d5c2f94f74d839d670b466641d5bd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 10:58:30.217325 29188 raft_consensus.cc:348] T 3c5443a90f7145a4bd5438b82f445dc2 P 82406c217cae4759ae4a406064a5cdfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82406c217cae4759ae4a406064a5cdfc" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 45531 } } peers { permanent_uuid: "117d5c2f94f74d839d670b466641d5bd" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 39819 } } peers { permanent_uuid: "7d809a95dd9f49448eb88d18dcd9b083" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38689 } }
I20260929 10:58:30.217492 29188 raft_consensus.cc:374] T 3c5443a90f7145a4bd5438b82f445dc2 P 82406c217cae4759ae4a406064a5cdfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:58:30.217536 29188 raft_consensus.cc:729] T 3c5443a90f7145a4bd5438b82f445dc2 P 82406c217cae4759ae4a406064a5cdfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82406c217cae4759ae4a406064a5cdfc, State: Initialized, Role: FOLLOWER
I20260929 10:58:30.217651 29188 consensus_queue.cc:260] T 3c5443a90f7145a4bd5438b82f445dc2 P 82406c217cae4759ae4a406064a5cdfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82406c217cae4759ae4a406064a5cdfc" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 45531 } } peers { permanent_uuid: "117d5c2f94f74d839d670b466641d5bd" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 39819 } } peers { permanent_uuid: "7d809a95dd9f49448eb88d18dcd9b083" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38689 } }
I20260929 10:58:30.217911 29188 ts_tablet_manager.cc:1428] T 3c5443a90f7145a4bd5438b82f445dc2 P 82406c217cae4759ae4a406064a5cdfc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 10:58:30.218047 28912 heartbeater.cc:503] Master 127.10.238.254:41903 was elected leader, sending a full tablet report...
I20260929 10:58:30.218281 29189 raft_consensus.cc:348] T 3c5443a90f7145a4bd5438b82f445dc2 P 7d809a95dd9f49448eb88d18dcd9b083 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82406c217cae4759ae4a406064a5cdfc" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 45531 } } peers { permanent_uuid: "117d5c2f94f74d839d670b466641d5bd" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 39819 } } peers { permanent_uuid: "7d809a95dd9f49448eb88d18dcd9b083" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38689 } }
I20260929 10:58:30.218438 29189 raft_consensus.cc:374] T 3c5443a90f7145a4bd5438b82f445dc2 P 7d809a95dd9f49448eb88d18dcd9b083 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:58:30.218473 29189 raft_consensus.cc:729] T 3c5443a90f7145a4bd5438b82f445dc2 P 7d809a95dd9f49448eb88d18dcd9b083 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d809a95dd9f49448eb88d18dcd9b083, State: Initialized, Role: FOLLOWER
I20260929 10:58:30.218603 29189 consensus_queue.cc:260] T 3c5443a90f7145a4bd5438b82f445dc2 P 7d809a95dd9f49448eb88d18dcd9b083 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82406c217cae4759ae4a406064a5cdfc" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 45531 } } peers { permanent_uuid: "117d5c2f94f74d839d670b466641d5bd" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 39819 } } peers { permanent_uuid: "7d809a95dd9f49448eb88d18dcd9b083" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38689 } }
I20260929 10:58:30.218868 29189 ts_tablet_manager.cc:1428] T 3c5443a90f7145a4bd5438b82f445dc2 P 7d809a95dd9f49448eb88d18dcd9b083: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260929 10:58:30.218904 29043 heartbeater.cc:503] Master 127.10.238.254:41903 was elected leader, sending a full tablet report...
I20260929 10:58:30.218961 29190 raft_consensus.cc:348] T 3c5443a90f7145a4bd5438b82f445dc2 P 117d5c2f94f74d839d670b466641d5bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82406c217cae4759ae4a406064a5cdfc" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 45531 } } peers { permanent_uuid: "117d5c2f94f74d839d670b466641d5bd" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 39819 } } peers { permanent_uuid: "7d809a95dd9f49448eb88d18dcd9b083" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38689 } }
I20260929 10:58:30.219107 29190 raft_consensus.cc:374] T 3c5443a90f7145a4bd5438b82f445dc2 P 117d5c2f94f74d839d670b466641d5bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:58:30.219153 29190 raft_consensus.cc:729] T 3c5443a90f7145a4bd5438b82f445dc2 P 117d5c2f94f74d839d670b466641d5bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 117d5c2f94f74d839d670b466641d5bd, State: Initialized, Role: FOLLOWER
I20260929 10:58:30.219275 29190 consensus_queue.cc:260] T 3c5443a90f7145a4bd5438b82f445dc2 P 117d5c2f94f74d839d670b466641d5bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82406c217cae4759ae4a406064a5cdfc" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 45531 } } peers { permanent_uuid: "117d5c2f94f74d839d670b466641d5bd" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 39819 } } peers { permanent_uuid: "7d809a95dd9f49448eb88d18dcd9b083" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38689 } }
I20260929 10:58:30.219558 29190 ts_tablet_manager.cc:1428] T 3c5443a90f7145a4bd5438b82f445dc2 P 117d5c2f94f74d839d670b466641d5bd: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260929 10:58:30.219653 29174 heartbeater.cc:503] Master 127.10.238.254:41903 was elected leader, sending a full tablet report...
W20260929 10:58:30.283618 29044 tablet.cc:2369] T 3c5443a90f7145a4bd5438b82f445dc2 P 7d809a95dd9f49448eb88d18dcd9b083: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 10:58:30.403549 28913 tablet.cc:2369] T 3c5443a90f7145a4bd5438b82f445dc2 P 82406c217cae4759ae4a406064a5cdfc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:58:30.415403 29195 raft_consensus.cc:482] T 3c5443a90f7145a4bd5438b82f445dc2 P 7d809a95dd9f49448eb88d18dcd9b083 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:58:30.415580 29195 raft_consensus.cc:504] T 3c5443a90f7145a4bd5438b82f445dc2 P 7d809a95dd9f49448eb88d18dcd9b083 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82406c217cae4759ae4a406064a5cdfc" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 45531 } } peers { permanent_uuid: "117d5c2f94f74d839d670b466641d5bd" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 39819 } } peers { permanent_uuid: "7d809a95dd9f49448eb88d18dcd9b083" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38689 } }
I20260929 10:58:30.415963 29195 leader_election.cc:288] T 3c5443a90f7145a4bd5438b82f445dc2 P 7d809a95dd9f49448eb88d18dcd9b083 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82406c217cae4759ae4a406064a5cdfc (127.10.238.193:45531), 117d5c2f94f74d839d670b466641d5bd (127.10.238.195:39819)
I20260929 10:58:30.420030 29187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:30.416485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:46445 (local address 127.10.238.195:39819)
0929 10:58:30.416543 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:30.416545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:30.416611 (+ 66us) server_negotiation.cc:407] Connection header received
0929 10:58:30.416710 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:30.416712 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:30.416720 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:30.416782 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:58:30.416917 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:30.417608 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:30.419030 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:30.419603 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:30.419688 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:30.419696 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:30.419902 (+ 206us) server_negotiation.cc:299] Negotiation successful
0929 10:58:30.419954 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 10:58:30.420030 29186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:30.416331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:47241 (local address 127.10.238.193:45531)
0929 10:58:30.416413 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:30.416415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:30.416427 (+ 12us) server_negotiation.cc:407] Connection header received
0929 10:58:30.416460 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:30.416461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:30.416474 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:30.416537 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:58:30.416719 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:30.417541 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:30.419030 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:30.419603 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:30.419688 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:30.419696 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:30.419903 (+ 207us) server_negotiation.cc:299] Negotiation successful
0929 10:58:30.419954 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 10:58:30.420019 29045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:30.416204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:45531 (local address 127.10.238.194:47241)
0929 10:58:30.416267 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:30.416281 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:58:30.416374 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:58:30.416548 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:58:30.416548 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:58:30.416551 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:58:30.416635 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:30.416637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:30.417621 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:30.417622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:30.418913 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:30.418916 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:30.419698 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:30.419699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:30.419774 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:30.419779 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 10:58:30.419842 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 10:58:30.419893 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":34}
I20260929 10:58:30.420027 29197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:30.416363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.195:39819 (local address 127.10.238.194:46445)
0929 10:58:30.416546 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:30.416556 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 10:58:30.416611 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:58:30.416791 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:58:30.416792 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:58:30.416793 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:58:30.416841 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:30.416843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:30.417688 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:30.417689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:30.418914 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:30.418916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:30.419698 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:30.419699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:30.419799 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:30.419802 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 10:58:30.419854 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 10:58:30.419893 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":15,"thread_start_us":112,"threads_started":1}
I20260929 10:58:30.420490 29128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5443a90f7145a4bd5438b82f445dc2" candidate_uuid: "7d809a95dd9f49448eb88d18dcd9b083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "117d5c2f94f74d839d670b466641d5bd" is_pre_election: true
I20260929 10:58:30.420485 28866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5443a90f7145a4bd5438b82f445dc2" candidate_uuid: "7d809a95dd9f49448eb88d18dcd9b083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82406c217cae4759ae4a406064a5cdfc" is_pre_election: true
I20260929 10:58:30.420657 28866 raft_consensus.cc:2454] T 3c5443a90f7145a4bd5438b82f445dc2 P 82406c217cae4759ae4a406064a5cdfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d809a95dd9f49448eb88d18dcd9b083 in term 0.
I20260929 10:58:30.420657 29128 raft_consensus.cc:2454] T 3c5443a90f7145a4bd5438b82f445dc2 P 117d5c2f94f74d839d670b466641d5bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d809a95dd9f49448eb88d18dcd9b083 in term 0.
I20260929 10:58:30.420903 28933 leader_election.cc:302] T 3c5443a90f7145a4bd5438b82f445dc2 P 7d809a95dd9f49448eb88d18dcd9b083 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d809a95dd9f49448eb88d18dcd9b083, 82406c217cae4759ae4a406064a5cdfc; no voters:
I20260929 10:58:30.421068 29195 raft_consensus.cc:2781] T 3c5443a90f7145a4bd5438b82f445dc2 P 7d809a95dd9f49448eb88d18dcd9b083 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:58:30.421152 29195 raft_consensus.cc:482] T 3c5443a90f7145a4bd5438b82f445dc2 P 7d809a95dd9f49448eb88d18dcd9b083 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:58:30.421171 29195 raft_consensus.cc:3037] T 3c5443a90f7145a4bd5438b82f445dc2 P 7d809a95dd9f49448eb88d18dcd9b083 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:58:30.421948 29195 raft_consensus.cc:504] T 3c5443a90f7145a4bd5438b82f445dc2 P 7d809a95dd9f49448eb88d18dcd9b083 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82406c217cae4759ae4a406064a5cdfc" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 45531 } } peers { permanent_uuid: "117d5c2f94f74d839d670b466641d5bd" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 39819 } } peers { permanent_uuid: "7d809a95dd9f49448eb88d18dcd9b083" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38689 } }
I20260929 10:58:30.422099 29195 leader_election.cc:288] T 3c5443a90f7145a4bd5438b82f445dc2 P 7d809a95dd9f49448eb88d18dcd9b083 [CANDIDATE]: Term 1 election: Requested vote from peers 82406c217cae4759ae4a406064a5cdfc (127.10.238.193:45531), 117d5c2f94f74d839d670b466641d5bd (127.10.238.195:39819)
I20260929 10:58:30.422298 28866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5443a90f7145a4bd5438b82f445dc2" candidate_uuid: "7d809a95dd9f49448eb88d18dcd9b083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82406c217cae4759ae4a406064a5cdfc"
I20260929 10:58:30.422298 29128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5443a90f7145a4bd5438b82f445dc2" candidate_uuid: "7d809a95dd9f49448eb88d18dcd9b083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "117d5c2f94f74d839d670b466641d5bd"
I20260929 10:58:30.422389 28866 raft_consensus.cc:3037] T 3c5443a90f7145a4bd5438b82f445dc2 P 82406c217cae4759ae4a406064a5cdfc [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:58:30.422389 29128 raft_consensus.cc:3037] T 3c5443a90f7145a4bd5438b82f445dc2 P 117d5c2f94f74d839d670b466641d5bd [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:58:30.423331 29128 raft_consensus.cc:2454] T 3c5443a90f7145a4bd5438b82f445dc2 P 117d5c2f94f74d839d670b466641d5bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d809a95dd9f49448eb88d18dcd9b083 in term 1.
I20260929 10:58:30.423331 28866 raft_consensus.cc:2454] T 3c5443a90f7145a4bd5438b82f445dc2 P 82406c217cae4759ae4a406064a5cdfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d809a95dd9f49448eb88d18dcd9b083 in term 1.
I20260929 10:58:30.423547 28933 leader_election.cc:302] T 3c5443a90f7145a4bd5438b82f445dc2 P 7d809a95dd9f49448eb88d18dcd9b083 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d809a95dd9f49448eb88d18dcd9b083, 82406c217cae4759ae4a406064a5cdfc; no voters:
I20260929 10:58:30.423650 29195 raft_consensus.cc:2781] T 3c5443a90f7145a4bd5438b82f445dc2 P 7d809a95dd9f49448eb88d18dcd9b083 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:58:30.423803 29195 raft_consensus.cc:686] T 3c5443a90f7145a4bd5438b82f445dc2 P 7d809a95dd9f49448eb88d18dcd9b083 [term 1 LEADER]: Becoming Leader. State: Replica: 7d809a95dd9f49448eb88d18dcd9b083, State: Running, Role: LEADER
I20260929 10:58:30.423914 29195 consensus_queue.cc:237] T 3c5443a90f7145a4bd5438b82f445dc2 P 7d809a95dd9f49448eb88d18dcd9b083 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82406c217cae4759ae4a406064a5cdfc" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 45531 } } peers { permanent_uuid: "117d5c2f94f74d839d670b466641d5bd" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 39819 } } peers { permanent_uuid: "7d809a95dd9f49448eb88d18dcd9b083" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38689 } }
I20260929 10:58:30.424690 28723 catalog_manager.cc:5495] T 3c5443a90f7145a4bd5438b82f445dc2 P 7d809a95dd9f49448eb88d18dcd9b083 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d809a95dd9f49448eb88d18dcd9b083 (127.10.238.194). New cstate: current_term: 1 leader_uuid: "7d809a95dd9f49448eb88d18dcd9b083" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82406c217cae4759ae4a406064a5cdfc" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 45531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "117d5c2f94f74d839d670b466641d5bd" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 39819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d809a95dd9f49448eb88d18dcd9b083" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 38689 } health_report { overall_health: HEALTHY } } }
W20260929 10:58:30.425515 29175 tablet.cc:2369] T 3c5443a90f7145a4bd5438b82f445dc2 P 117d5c2f94f74d839d670b466641d5bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:58:30.438928 29184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:30.436419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51976 (local address 127.10.238.194:38689)
0929 10:58:30.436485 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:30.436487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:30.436496 (+ 9us) server_negotiation.cc:407] Connection header received
0929 10:58:30.436541 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:30.436542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:30.436560 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:30.436641 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:58:30.436794 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:30.437358 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:30.438104 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:30.438310 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:30.438387 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:30.438559 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:58:30.438666 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:58:30.438761 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:30.438798 (+ 37us) server_negotiation.cc:299] Negotiation successful
0929 10:58:30.438821 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 10:58:30.440972 29128 raft_consensus.cc:1261] T 3c5443a90f7145a4bd5438b82f445dc2 P 117d5c2f94f74d839d670b466641d5bd [term 1 FOLLOWER]: Refusing update from remote peer 7d809a95dd9f49448eb88d18dcd9b083: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:58:30.440973 28866 raft_consensus.cc:1261] T 3c5443a90f7145a4bd5438b82f445dc2 P 82406c217cae4759ae4a406064a5cdfc [term 1 FOLLOWER]: Refusing update from remote peer 7d809a95dd9f49448eb88d18dcd9b083: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:58:30.441414 29195 consensus_queue.cc:1035] T 3c5443a90f7145a4bd5438b82f445dc2 P 7d809a95dd9f49448eb88d18dcd9b083 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82406c217cae4759ae4a406064a5cdfc" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 45531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:58:30.441576 29198 consensus_queue.cc:1035] T 3c5443a90f7145a4bd5438b82f445dc2 P 7d809a95dd9f49448eb88d18dcd9b083 [LEADER]: Connected to new peer: Peer: permanent_uuid: "117d5c2f94f74d839d670b466641d5bd" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 39819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:58:30.446904 28722 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:58:30.446995 28722 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:58:30.447775 28722 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 962e8d5532774e1fb590d7f41f491c39: Sending DeleteTablet for 3 replicas of tablet 3c5443a90f7145a4bd5438b82f445dc2
I20260929 10:58:30.448134 28846 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c5443a90f7145a4bd5438b82f445dc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:58:30 UTC) from {username='slave'} at 127.0.0.1:51820
I20260929 10:58:30.448215 29108 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c5443a90f7145a4bd5438b82f445dc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:58:30 UTC) from {username='slave'} at 127.0.0.1:40384
I20260929 10:58:30.448386 28977 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c5443a90f7145a4bd5438b82f445dc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:58:30 UTC) from {username='slave'} at 127.0.0.1:51962
I20260929 10:58:30.448406 29211 tablet_replica.cc:330] T 3c5443a90f7145a4bd5438b82f445dc2 P 82406c217cae4759ae4a406064a5cdfc: stopping tablet replica
I20260929 10:58:30.448513 29211 raft_consensus.cc:2229] T 3c5443a90f7145a4bd5438b82f445dc2 P 82406c217cae4759ae4a406064a5cdfc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:58:30.448611 29212 tablet_replica.cc:330] T 3c5443a90f7145a4bd5438b82f445dc2 P 117d5c2f94f74d839d670b466641d5bd: stopping tablet replica
I20260929 10:58:30.448637 29211 raft_consensus.cc:2258] T 3c5443a90f7145a4bd5438b82f445dc2 P 82406c217cae4759ae4a406064a5cdfc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:58:30.448833 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 28784
I20260929 10:58:30.448796 29212 raft_consensus.cc:2229] T 3c5443a90f7145a4bd5438b82f445dc2 P 117d5c2f94f74d839d670b466641d5bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:58:30.449558 29212 raft_consensus.cc:2258] T 3c5443a90f7145a4bd5438b82f445dc2 P 117d5c2f94f74d839d670b466641d5bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:58:30.450192 29213 tablet_replica.cc:330] T 3c5443a90f7145a4bd5438b82f445dc2 P 7d809a95dd9f49448eb88d18dcd9b083: stopping tablet replica
I20260929 10:58:30.450299 29213 raft_consensus.cc:2229] T 3c5443a90f7145a4bd5438b82f445dc2 P 7d809a95dd9f49448eb88d18dcd9b083 [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:58:30.450415 29213 raft_consensus.cc:2258] T 3c5443a90f7145a4bd5438b82f445dc2 P 7d809a95dd9f49448eb88d18dcd9b083 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:58:30.450851 29213 ts_tablet_manager.cc:1905] T 3c5443a90f7145a4bd5438b82f445dc2 P 7d809a95dd9f49448eb88d18dcd9b083: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:58:30.451270 29212 ts_tablet_manager.cc:1905] T 3c5443a90f7145a4bd5438b82f445dc2 P 117d5c2f94f74d839d670b466641d5bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:58:30.452533 29213 ts_tablet_manager.cc:1918] T 3c5443a90f7145a4bd5438b82f445dc2 P 7d809a95dd9f49448eb88d18dcd9b083: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:58:30.452630 29213 log.cc:1196] T 3c5443a90f7145a4bd5438b82f445dc2 P 7d809a95dd9f49448eb88d18dcd9b083: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/ts-1/wal/wals/3c5443a90f7145a4bd5438b82f445dc2
I20260929 10:58:30.452958 29213 ts_tablet_manager.cc:1939] T 3c5443a90f7145a4bd5438b82f445dc2 P 7d809a95dd9f49448eb88d18dcd9b083: Deleting consensus metadata
I20260929 10:58:30.453351 29212 ts_tablet_manager.cc:1918] T 3c5443a90f7145a4bd5438b82f445dc2 P 117d5c2f94f74d839d670b466641d5bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:58:30.453426 29212 log.cc:1196] T 3c5443a90f7145a4bd5438b82f445dc2 P 117d5c2f94f74d839d670b466641d5bd: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790679274510869-11195-0/minicluster-data/ts-2/wal/wals/3c5443a90f7145a4bd5438b82f445dc2
I20260929 10:58:30.453397 28709 catalog_manager.cc:4841] TS 7d809a95dd9f49448eb88d18dcd9b083 (127.10.238.194:38689): tablet 3c5443a90f7145a4bd5438b82f445dc2 (table test-table [id=2dfd08392d754908b45617936ddeacee]) successfully deleted
I20260929 10:58:30.453668 29212 ts_tablet_manager.cc:1939] T 3c5443a90f7145a4bd5438b82f445dc2 P 117d5c2f94f74d839d670b466641d5bd: Deleting consensus metadata
I20260929 10:58:30.454393 28711 catalog_manager.cc:4841] TS 117d5c2f94f74d839d670b466641d5bd (127.10.238.195:39819): tablet 3c5443a90f7145a4bd5438b82f445dc2 (table test-table [id=2dfd08392d754908b45617936ddeacee]) successfully deleted
W20260929 10:58:30.454988 28711 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.238.193:45531 (error 108)
I20260929 10:58:30.455185 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 28915
I20260929 10:58:30.458014 29185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:30.457846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:45531 (local address 127.0.0.1:51834)
0929 10:58:30.457916 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:30.457970 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.193:45531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 10:58:30.458175 28711 catalog_manager.cc:4584] TS 82406c217cae4759ae4a406064a5cdfc (127.10.238.193:45531): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3c5443a90f7145a4bd5438b82f445dc2: Network error: Client connection negotiation failed: client connection to 127.10.238.193:45531: connect: Connection refused (error 111)
I20260929 10:58:30.460510 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 29046
I20260929 10:58:30.465771 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 28693
2026-09-29T10:58:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (895 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:58:30 dist-test-slave-8nrt krb5kdc[29217](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:58:30 dist-test-slave-8nrt krb5kdc[29217](info): set up 2 sockets
Sep 29 10:58:30 dist-test-slave-8nrt krb5kdc[29217](info): commencing operation
krb5kdc: starting...
W20260929 10:58:31.529327 11195 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Sep 29 10:58:31 dist-test-slave-8nrt krb5kdc[29217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:58:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:58:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
I20260929 10:58:31.648751 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:45063
--webserver_interface=127.10.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:32943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:45063
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:58:31.730649 29230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:58:31.730858 29230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:58:31.730881 29230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:58:31.732064 29230 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:58:31.732106 29230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:58:31.732121 29230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:58:31.732134 29230 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:58:31.732187 29230 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:58:31.733645 29230 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:32943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:45063
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:45063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:58:31.733904 29230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:58:31.734138 29230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:58:31.739141 29235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:31.739192 29230 server_base.cc:1034] running on GCE node
W20260929 10:58:31.739123 29238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:31.739225 29236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:31.739629 29230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:58:31.739993 29230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:58:31.741159 29230 hybrid_clock.cc:648] HybridClock initialized: now 1790679511741146 us; error 44 us; skew 500 ppm
Sep 29 10:58:31 dist-test-slave-8nrt krb5kdc[29217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679511, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:58:31.743809 29230 init.cc:377] Logged in from keytab as kudu/127.10.238.254@KRBTEST.COM (short username kudu)
I20260929 10:58:31.744251 29230 webserver.cc:460] Webserver started at http://127.10.238.254:40255/ using document root <none> and password file <none>
I20260929 10:58:31.744488 29230 fs_manager.cc:359] Metadata directory not provided
I20260929 10:58:31.744558 29230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:58:31.744705 29230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:58:31.745826 29230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/master-0/data/instance:
uuid: "7b8fc327412c412a98bc589b3d13db19"
format_stamp: "Formatted at 2026-09-29 10:58:31 on dist-test-slave-8nrt"
server_key: "a87250fc05cb3bb4211b21cfd2a4b519"
server_key_iv: "fa653d80c209517fdd18341bec10905c"
server_key_version: "encryptionkey@0"
I20260929 10:58:31.746222 29230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/master-0/wal/instance:
uuid: "7b8fc327412c412a98bc589b3d13db19"
format_stamp: "Formatted at 2026-09-29 10:58:31 on dist-test-slave-8nrt"
server_key: "a87250fc05cb3bb4211b21cfd2a4b519"
server_key_iv: "fa653d80c209517fdd18341bec10905c"
server_key_version: "encryptionkey@0"
I20260929 10:58:31.747555 29230 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 10:58:31.748281 29245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:31.748447 29230 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:58:31.748553 29230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "7b8fc327412c412a98bc589b3d13db19"
format_stamp: "Formatted at 2026-09-29 10:58:31 on dist-test-slave-8nrt"
server_key: "a87250fc05cb3bb4211b21cfd2a4b519"
server_key_iv: "fa653d80c209517fdd18341bec10905c"
server_key_version: "encryptionkey@0"
I20260929 10:58:31.748631 29230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:58:31.756973 29230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:58:31.759310 29230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:58:31.759476 29230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:58:31.763516 29230 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.254:45063
I20260929 10:58:31.763551 29297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.254:45063 every 8 connection(s)
I20260929 10:58:31.763899 29230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:58:31.764735 29298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:58:31.767059 29298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b8fc327412c412a98bc589b3d13db19: Bootstrap starting.
I20260929 10:58:31.767642 29298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b8fc327412c412a98bc589b3d13db19: Neither blocks nor log segments found. Creating new log.
I20260929 10:58:31.767939 29298 log.cc:824] T 00000000000000000000000000000000 P 7b8fc327412c412a98bc589b3d13db19: Log is configured to *not* fsync() on all Append() calls
I20260929 10:58:31.768615 29298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b8fc327412c412a98bc589b3d13db19: No bootstrap required, opened a new log
I20260929 10:58:31.770274 29298 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b8fc327412c412a98bc589b3d13db19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b8fc327412c412a98bc589b3d13db19" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 45063 } }
I20260929 10:58:31.770408 29298 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b8fc327412c412a98bc589b3d13db19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:58:31.770430 29298 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b8fc327412c412a98bc589b3d13db19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b8fc327412c412a98bc589b3d13db19, State: Initialized, Role: FOLLOWER
I20260929 10:58:31.770624 29298 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b8fc327412c412a98bc589b3d13db19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b8fc327412c412a98bc589b3d13db19" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 45063 } }
I20260929 10:58:31.770704 29298 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b8fc327412c412a98bc589b3d13db19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:58:31.770754 29298 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b8fc327412c412a98bc589b3d13db19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:58:31.770807 29298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b8fc327412c412a98bc589b3d13db19 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:58:31.771317 29298 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b8fc327412c412a98bc589b3d13db19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b8fc327412c412a98bc589b3d13db19" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 45063 } }
I20260929 10:58:31.771430 29298 leader_election.cc:302] T 00000000000000000000000000000000 P 7b8fc327412c412a98bc589b3d13db19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b8fc327412c412a98bc589b3d13db19; no voters:
I20260929 10:58:31.771605 29298 leader_election.cc:288] T 00000000000000000000000000000000 P 7b8fc327412c412a98bc589b3d13db19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:58:31.771646 29301 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b8fc327412c412a98bc589b3d13db19 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:58:31.771834 29301 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b8fc327412c412a98bc589b3d13db19 [term 1 LEADER]: Becoming Leader. State: Replica: 7b8fc327412c412a98bc589b3d13db19, State: Running, Role: LEADER
I20260929 10:58:31.771854 29298 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b8fc327412c412a98bc589b3d13db19 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:58:31.771932 29301 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b8fc327412c412a98bc589b3d13db19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b8fc327412c412a98bc589b3d13db19" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 45063 } }
I20260929 10:58:31.772310 29303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b8fc327412c412a98bc589b3d13db19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b8fc327412c412a98bc589b3d13db19. Latest consensus state: current_term: 1 leader_uuid: "7b8fc327412c412a98bc589b3d13db19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b8fc327412c412a98bc589b3d13db19" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 45063 } } }
I20260929 10:58:31.772397 29303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b8fc327412c412a98bc589b3d13db19 [sys.catalog]: This master's current role is: LEADER
I20260929 10:58:31.772502 29302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b8fc327412c412a98bc589b3d13db19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b8fc327412c412a98bc589b3d13db19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b8fc327412c412a98bc589b3d13db19" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 45063 } } }
I20260929 10:58:31.772594 29302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b8fc327412c412a98bc589b3d13db19 [sys.catalog]: This master's current role is: LEADER
I20260929 10:58:31.772637 29310 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:58:31.773175 29310 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:58:31.773448 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 29230
I20260929 10:58:31.773566 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/master-0/wal/instance
I20260929 10:58:31.773789 11195 external_mini_cluster.cc:1428] Setting key 82587ad62fe1119e0b310be5f88e9f33
I20260929 10:58:31.775614 29310 catalog_manager.cc:1347] Generated new cluster ID: 41d540368d254eb2b66f7a4ab725f247
I20260929 10:58:31.775667 29310 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 10:58:31 dist-test-slave-8nrt krb5kdc[29217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:58:31.784027 29320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:31.774218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35500 (local address 127.10.238.254:45063)
0929 10:58:31.774761 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:31.774769 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:31.774804 (+ 35us) server_negotiation.cc:407] Connection header received
0929 10:58:31.774913 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:31.774920 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:31.774974 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:31.775120 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:31.776032 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:31.776973 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:31.777813 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:31.778247 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:31.778390 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:31.780690 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:31.780694 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:31.780694 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:58:31.780706 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:31.782415 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:31.782804 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:31.782807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:31.782807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:31.782905 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:31.783118 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:31.783119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:31.783119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:31.783354 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:31.783518 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:31.783615 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 10:58:31.783864 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":79,"threads_started":1}
I20260929 10:58:31.791816 29310 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:58:31.792493 29310 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:58:31.802284 29310 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b8fc327412c412a98bc589b3d13db19: Generated new TSK 0
I20260929 10:58:31.802480 29310 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.238.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab.
Entry for principal kudu/127.10.238.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab.
I20260929 10:58:31.836500 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.193:0
--local_ip_for_outbound_sockets=127.10.238.193
--webserver_interface=127.10.238.193
--webserver_port=0
--tserver_master_addrs=127.10.238.254:45063
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab
--principal=kudu/127.10.238.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:32943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:58:31.930207 29324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:58:31.930411 29324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:58:31.930434 29324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:58:31.931604 29324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:58:31.931646 29324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:58:31.931672 29324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.193
I20260929 10:58:31.933102 29324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:32943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab
--principal=kudu/127.10.238.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.238.193
--webserver_port=0
--tserver_master_addrs=127.10.238.254:45063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.193
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:58:31.933398 29324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:58:31.933650 29324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:58:31.939047 29330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:31.939064 29324 server_base.cc:1034] running on GCE node
W20260929 10:58:31.939035 29329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:31.939035 29332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:31.939524 29324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:58:31.939733 29324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:58:31.940912 29324 hybrid_clock.cc:648] HybridClock initialized: now 1790679511940910 us; error 45 us; skew 500 ppm
Sep 29 10:58:31 dist-test-slave-8nrt krb5kdc[29217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679511, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:58:31.943463 29324 init.cc:377] Logged in from keytab as kudu/127.10.238.193@KRBTEST.COM (short username kudu)
I20260929 10:58:31.943897 29324 webserver.cc:460] Webserver started at http://127.10.238.193:44201/ using document root <none> and password file <none>
I20260929 10:58:31.944115 29324 fs_manager.cc:359] Metadata directory not provided
I20260929 10:58:31.944182 29324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:58:31.944335 29324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:58:31.945466 29324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/ts-0/data/instance:
uuid: "a0706d043bb148f9b532fcdd8c52ce2a"
format_stamp: "Formatted at 2026-09-29 10:58:31 on dist-test-slave-8nrt"
server_key: "515e02820bc00f3d485b29473645d128"
server_key_iv: "c7bc03cfa1fd2bf0f78faa1953aa806f"
server_key_version: "encryptionkey@0"
I20260929 10:58:31.945829 29324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/ts-0/wal/instance:
uuid: "a0706d043bb148f9b532fcdd8c52ce2a"
format_stamp: "Formatted at 2026-09-29 10:58:31 on dist-test-slave-8nrt"
server_key: "515e02820bc00f3d485b29473645d128"
server_key_iv: "c7bc03cfa1fd2bf0f78faa1953aa806f"
server_key_version: "encryptionkey@0"
I20260929 10:58:31.947147 29324 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:58:31.947889 29339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:31.948062 29324 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:58:31.948160 29324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/ts-0/wal
uuid: "a0706d043bb148f9b532fcdd8c52ce2a"
format_stamp: "Formatted at 2026-09-29 10:58:31 on dist-test-slave-8nrt"
server_key: "515e02820bc00f3d485b29473645d128"
server_key_iv: "c7bc03cfa1fd2bf0f78faa1953aa806f"
server_key_version: "encryptionkey@0"
I20260929 10:58:31.948252 29324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:58:31.981814 29324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:58:31.983791 29324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:58:31.983961 29324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:58:31.984248 29324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:58:31.984578 29324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:58:31.984633 29324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:31.984666 29324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:58:31.984692 29324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:31.990496 29324 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.193:37817
I20260929 10:58:31.990546 29452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.193:37817 every 8 connection(s)
I20260929 10:58:31.990840 29324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
I20260929 10:58:31.992267 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 29324
I20260929 10:58:31.992385 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/ts-0/wal/instance
I20260929 10:58:31.992602 11195 external_mini_cluster.cc:1428] Setting key 7b7428a821ea25176271036d1c6ffb02
Sep 29 10:58:31 dist-test-slave-8nrt krb5kdc[29217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679511, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.193@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.238.194@KRBTEST.COM; defaulting to no policy
I20260929 10:58:32.001158 29320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:31.991495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:60021 (local address 127.10.238.254:45063)
0929 10:58:31.991540 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:31.991542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:31.991785 (+ 243us) server_negotiation.cc:407] Connection header received
0929 10:58:31.992252 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:31.992254 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:31.992270 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:31.992348 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:31.993585 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:31.994382 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:31.995420 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:31.995766 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:31.995847 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:31.998293 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:31.998294 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:31.998295 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:58:31.998302 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:31.999734 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:32.000167 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:32.000169 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:32.000169 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:32.000231 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:32.000502 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:32.000503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:32.000504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:32.000618 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:32.000697 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:32.000973 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 10:58:32.001071 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 10:58:32.001428 29455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:31.991462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:45063 (local address 127.10.238.193:60021)
0929 10:58:31.991644 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:31.991694 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 10:58:31.992154 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:58:31.992459 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:58:31.992463 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:58:31.993164 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:58:31.993485 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:31.993491 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:31.994483 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:31.994484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:31.995319 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:31.995322 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:31.995857 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:31.995857 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:31.996006 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:31.996489 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:58:31.996498 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:58:31.998176 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:58:31.999850 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:58:31.999858 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:58:31.999860 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:58:32.000086 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:58:32.000307 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:58:32.000308 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:58:32.000309 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:58:32.000429 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:58:32.000689 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:58:32.000696 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:58:32.000837 (+ 141us) client_negotiation.cc:769] Sending connection context
0929 10:58:32.000951 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 10:58:32.001113 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":99,"threads_started":1}
I20260929 10:58:32.001924 29453 heartbeater.cc:347] Connected to a master server at 127.10.238.254:45063
I20260929 10:58:32.002023 29453 heartbeater.cc:464] Registering TS with master...
I20260929 10:58:32.002233 29453 heartbeater.cc:511] Master 127.10.238.254:45063 requested a full tablet report, sending...
I20260929 10:58:32.002792 29262 ts_manager.cc:194] Registered new tserver with Master: a0706d043bb148f9b532fcdd8c52ce2a (127.10.238.193:37817)
I20260929 10:58:32.003409 29262 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.193@KRBTEST.COM'} at 127.10.238.193:60021
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab.
Entry for principal kudu/127.10.238.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab.
I20260929 10:58:32.017836 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.194:0
--local_ip_for_outbound_sockets=127.10.238.194
--webserver_interface=127.10.238.194
--webserver_port=0
--tserver_master_addrs=127.10.238.254:45063
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab
--principal=kudu/127.10.238.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:32943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:58:32.099002 29458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:58:32.099252 29458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:58:32.099293 29458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:58:32.100440 29458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:58:32.100522 29458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:58:32.100569 29458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.194
I20260929 10:58:32.102108 29458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:32943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab
--principal=kudu/127.10.238.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.238.194
--webserver_port=0
--tserver_master_addrs=127.10.238.254:45063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.194
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:58:32.102372 29458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:58:32.102603 29458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:58:32.107581 29463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:32.107689 29464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:32.107671 29466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:32.107887 29458 server_base.cc:1034] running on GCE node
I20260929 10:58:32.108127 29458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:58:32.108356 29458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:58:32.109552 29458 hybrid_clock.cc:648] HybridClock initialized: now 1790679512109536 us; error 29 us; skew 500 ppm
Sep 29 10:58:32 dist-test-slave-8nrt krb5kdc[29217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679512, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:58:32.112082 29458 init.cc:377] Logged in from keytab as kudu/127.10.238.194@KRBTEST.COM (short username kudu)
I20260929 10:58:32.112555 29458 webserver.cc:460] Webserver started at http://127.10.238.194:39811/ using document root <none> and password file <none>
I20260929 10:58:32.112793 29458 fs_manager.cc:359] Metadata directory not provided
I20260929 10:58:32.112864 29458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:58:32.113013 29458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:58:32.114096 29458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/ts-1/data/instance:
uuid: "c31719669d5343d582466e84031d62ea"
format_stamp: "Formatted at 2026-09-29 10:58:32 on dist-test-slave-8nrt"
server_key: "25807500c925de16278898103e493a33"
server_key_iv: "a27fb602ac19fd2234d20a5e07a57b01"
server_key_version: "encryptionkey@0"
I20260929 10:58:32.114467 29458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/ts-1/wal/instance:
uuid: "c31719669d5343d582466e84031d62ea"
format_stamp: "Formatted at 2026-09-29 10:58:32 on dist-test-slave-8nrt"
server_key: "25807500c925de16278898103e493a33"
server_key_iv: "a27fb602ac19fd2234d20a5e07a57b01"
server_key_version: "encryptionkey@0"
I20260929 10:58:32.115880 29458 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 10:58:32.116673 29473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:32.116856 29458 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 10:58:32.116956 29458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/ts-1/wal
uuid: "c31719669d5343d582466e84031d62ea"
format_stamp: "Formatted at 2026-09-29 10:58:32 on dist-test-slave-8nrt"
server_key: "25807500c925de16278898103e493a33"
server_key_iv: "a27fb602ac19fd2234d20a5e07a57b01"
server_key_version: "encryptionkey@0"
I20260929 10:58:32.117043 29458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:58:32.141607 29458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:58:32.143640 29458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:58:32.143812 29458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:58:32.144048 29458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:58:32.144369 29458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:58:32.144426 29458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:32.144497 29458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:58:32.144528 29458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:32.150744 29458 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.194:35333
I20260929 10:58:32.150842 29586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.194:35333 every 8 connection(s)
I20260929 10:58:32.151161 29458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
I20260929 10:58:32.152686 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 29458
I20260929 10:58:32.152814 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/ts-1/wal/instance
I20260929 10:58:32.153049 11195 external_mini_cluster.cc:1428] Setting key 0faa5f2ae30ff43c0da2b23a14631019
Sep 29 10:58:32 dist-test-slave-8nrt krb5kdc[29217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679512, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.194@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.238.195@KRBTEST.COM; defaulting to no policy
I20260929 10:58:32.162288 29320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:32.152046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:48577 (local address 127.10.238.254:45063)
0929 10:58:32.152142 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:32.152145 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:32.152349 (+ 204us) server_negotiation.cc:407] Connection header received
0929 10:58:32.153255 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:32.153256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:32.153296 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:32.153366 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:32.154714 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:32.155233 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:32.156253 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:32.156458 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:32.156546 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:32.159256 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:32.159257 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:32.159258 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:58:32.159264 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:32.160661 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:32.161144 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:32.161145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:32.161145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:32.161203 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:32.161527 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:32.161527 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:32.161528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:32.161630 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:32.161682 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:32.162029 (+ 347us) server_negotiation.cc:299] Negotiation successful
0929 10:58:32.162143 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 10:58:32.162461 29589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:32.151936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:45063 (local address 127.10.238.194:48577)
0929 10:58:32.152198 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:32.152245 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 10:58:32.153065 (+ 820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:58:32.153470 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:58:32.153474 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:58:32.154328 (+ 854us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:58:32.154615 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:32.154620 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:32.155330 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:32.155331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:32.156133 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:32.156135 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:32.156557 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:32.156558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:32.156698 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:32.157317 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:58:32.157325 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:58:32.159149 (+ 1824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:58:32.160753 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:58:32.160762 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:58:32.160765 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:58:32.161070 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:58:32.161313 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:58:32.161314 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:58:32.161314 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:58:32.161455 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:58:32.161701 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:58:32.161708 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:58:32.161888 (+ 180us) client_negotiation.cc:769] Sending connection context
0929 10:58:32.162015 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 10:58:32.162226 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":103,"threads_started":1}
I20260929 10:58:32.162997 29587 heartbeater.cc:347] Connected to a master server at 127.10.238.254:45063
I20260929 10:58:32.163080 29587 heartbeater.cc:464] Registering TS with master...
I20260929 10:58:32.163225 29587 heartbeater.cc:511] Master 127.10.238.254:45063 requested a full tablet report, sending...
I20260929 10:58:32.163527 29262 ts_manager.cc:194] Registered new tserver with Master: c31719669d5343d582466e84031d62ea (127.10.238.194:35333)
I20260929 10:58:32.163971 29262 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.194@KRBTEST.COM'} at 127.10.238.194:48577
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab.
Entry for principal kudu/127.10.238.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab.
I20260929 10:58:32.180050 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.195:0
--local_ip_for_outbound_sockets=127.10.238.195
--webserver_interface=127.10.238.195
--webserver_port=0
--tserver_master_addrs=127.10.238.254:45063
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab
--principal=kudu/127.10.238.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:32943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:58:32.262041 29592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:58:32.262260 29592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:58:32.262283 29592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:58:32.263542 29592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:58:32.263585 29592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:58:32.263622 29592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.195
I20260929 10:58:32.265013 29592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:32943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab
--principal=kudu/127.10.238.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.238.195
--webserver_port=0
--tserver_master_addrs=127.10.238.254:45063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.195
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:58:32.265198 29592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:58:32.265477 29592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:58:32.270355 29598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:32.270458 29600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:32.270346 29597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:32.270599 29592 server_base.cc:1034] running on GCE node
I20260929 10:58:32.270922 29592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:58:32.271198 29592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:58:32.272414 29592 hybrid_clock.cc:648] HybridClock initialized: now 1790679512272394 us; error 51 us; skew 500 ppm
Sep 29 10:58:32 dist-test-slave-8nrt krb5kdc[29217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679512, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:58:32.275218 29592 init.cc:377] Logged in from keytab as kudu/127.10.238.195@KRBTEST.COM (short username kudu)
I20260929 10:58:32.275669 29592 webserver.cc:460] Webserver started at http://127.10.238.195:33033/ using document root <none> and password file <none>
I20260929 10:58:32.275906 29592 fs_manager.cc:359] Metadata directory not provided
I20260929 10:58:32.275978 29592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:58:32.276144 29592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:58:32.277356 29592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/ts-2/data/instance:
uuid: "483d50e68119450fab798f37ebcdead0"
format_stamp: "Formatted at 2026-09-29 10:58:32 on dist-test-slave-8nrt"
server_key: "e49a8eb0fa09d988a7656e2474824886"
server_key_iv: "736f77cb225f7b8cd058230f2df738c6"
server_key_version: "encryptionkey@0"
I20260929 10:58:32.277747 29592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/ts-2/wal/instance:
uuid: "483d50e68119450fab798f37ebcdead0"
format_stamp: "Formatted at 2026-09-29 10:58:32 on dist-test-slave-8nrt"
server_key: "e49a8eb0fa09d988a7656e2474824886"
server_key_iv: "736f77cb225f7b8cd058230f2df738c6"
server_key_version: "encryptionkey@0"
I20260929 10:58:32.279158 29592 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 10:58:32.279971 29607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:32.280253 29592 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 10:58:32.280361 29592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/ts-2/wal
uuid: "483d50e68119450fab798f37ebcdead0"
format_stamp: "Formatted at 2026-09-29 10:58:32 on dist-test-slave-8nrt"
server_key: "e49a8eb0fa09d988a7656e2474824886"
server_key_iv: "736f77cb225f7b8cd058230f2df738c6"
server_key_version: "encryptionkey@0"
I20260929 10:58:32.280440 29592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:58:32.301546 29592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:58:32.303857 29592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:58:32.304039 29592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:58:32.304293 29592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:58:32.304625 29592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:58:32.304683 29592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:32.304757 29592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:58:32.304788 29592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:32.310752 29592 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.195:43107
I20260929 10:58:32.310833 29720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.195:43107 every 8 connection(s)
I20260929 10:58:32.311169 29592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
I20260929 10:58:32.315110 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 29592
I20260929 10:58:32.315246 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/ts-2/wal/instance
I20260929 10:58:32.315441 11195 external_mini_cluster.cc:1428] Setting key ceb0a49ad023f3a28d4f440e5ea862ac
Sep 29 10:58:32 dist-test-slave-8nrt krb5kdc[29217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679512, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.195@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:58:32.320884 29320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:32.311893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:54645 (local address 127.10.238.254:45063)
0929 10:58:32.311962 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:32.311964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:32.312156 (+ 192us) server_negotiation.cc:407] Connection header received
0929 10:58:32.312699 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:32.312700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:32.312710 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:32.312773 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:32.313841 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:32.314330 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:32.315348 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:32.315635 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:32.315728 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:32.318236 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:32.318237 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:32.318237 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:58:32.318244 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:32.319493 (+ 1249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:32.319997 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:32.319998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:32.319999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:32.320055 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:32.320294 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:32.320294 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:32.320295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:32.320395 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:32.320444 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:32.320679 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 10:58:32.320806 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 10:58:32.321121 29723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:32.311855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:45063 (local address 127.10.238.195:54645)
0929 10:58:32.312016 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:32.312061 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 10:58:32.312588 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:58:32.312896 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:58:32.312900 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:58:32.313552 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:58:32.313756 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:32.313761 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:32.314412 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:32.314412 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:32.315202 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:32.315204 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:32.315753 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:32.315754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:32.315916 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:32.316484 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:58:32.316496 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:58:32.318151 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:58:32.319594 (+ 1443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:58:32.319600 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:58:32.319602 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:58:32.319927 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:58:32.320118 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:58:32.320119 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:58:32.320119 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:58:32.320231 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:58:32.320450 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:58:32.320455 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:58:32.320568 (+ 113us) client_negotiation.cc:769] Sending connection context
0929 10:58:32.320651 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 10:58:32.320847 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":90,"threads_started":1}
I20260929 10:58:32.321686 29721 heartbeater.cc:347] Connected to a master server at 127.10.238.254:45063
I20260929 10:58:32.321798 29721 heartbeater.cc:464] Registering TS with master...
I20260929 10:58:32.321972 29721 heartbeater.cc:511] Master 127.10.238.254:45063 requested a full tablet report, sending...
I20260929 10:58:32.322332 29262 ts_manager.cc:194] Registered new tserver with Master: 483d50e68119450fab798f37ebcdead0 (127.10.238.195:43107)
I20260929 10:58:32.322710 29262 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.195@KRBTEST.COM'} at 127.10.238.195:54645
I20260929 10:58:32.327327 11195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:58:32.334496 29320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:32.328616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35506 (local address 127.10.238.254:45063)
0929 10:58:32.328663 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:32.328665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:32.328767 (+ 102us) server_negotiation.cc:407] Connection header received
0929 10:58:32.328817 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:32.328818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:32.328826 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:32.328878 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:32.329449 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:32.329909 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:32.330586 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:32.330759 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:32.330806 (+ 47us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:32.332005 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:32.332007 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:32.332007 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:58:32.332013 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:32.333488 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:32.333840 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:32.333841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:32.333841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:32.333891 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:32.334083 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:32.334084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:32.334084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:32.334211 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:32.334289 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:32.334390 (+ 101us) server_negotiation.cc:299] Negotiation successful
0929 10:58:32.334436 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 10:58:32.336272 29262 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35506:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:58:32.337963 29262 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:58:32.344491 29734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:32.340473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34214 (local address 127.10.238.194:35333)
0929 10:58:32.340780 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:32.340785 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:32.340795 (+ 10us) server_negotiation.cc:407] Connection header received
0929 10:58:32.340838 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:32.340841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:32.340857 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:32.340942 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:58:32.341162 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:32.342035 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:32.343356 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:32.344047 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:32.344152 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:32.344215 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:32.344365 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 10:58:32.344408 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":89,"threads_started":1}
I20260929 10:58:32.344478 29732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:32.340351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.194:35333 (local address 127.0.0.1:34214)
0929 10:58:32.340535 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:32.340587 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 10:58:32.340658 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:58:32.341004 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:58:32.341007 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:58:32.341009 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:58:32.341073 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:32.341075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:32.342147 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:32.342148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:32.343256 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:32.343259 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:32.344153 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:32.344154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:32.344233 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:32.344246 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 10:58:32.344345 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 10:58:32.344402 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":87,"threads_started":1}
I20260929 10:58:32.344926 29733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:32.340351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:37817 (local address 127.0.0.1:52298)
0929 10:58:32.340710 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:32.340722 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:58:32.340793 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:58:32.341074 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:58:32.341075 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:58:32.341078 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:58:32.341150 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:32.341152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:32.342672 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:32.342673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:32.343783 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:32.343785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:32.344697 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:32.344698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:32.344748 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:32.344756 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 10:58:32.344792 (+ 36us) client_negotiation.cc:240] Negotiation successful
0929 10:58:32.344840 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":35,"thread_start_us":65,"threads_started":1}
I20260929 10:58:32.344988 29737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:32.340797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44454 (local address 127.10.238.195:43107)
0929 10:58:32.341027 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:32.341032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:32.341042 (+ 10us) server_negotiation.cc:407] Connection header received
0929 10:58:32.341103 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:32.341105 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:32.341123 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:32.341206 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:58:32.341428 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:32.342536 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:32.343761 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:32.344490 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:32.344585 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:32.344640 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:32.344857 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 10:58:32.344908 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":107,"threads_started":1}
I20260929 10:58:32.344926 29736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:32.340565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.195:43107 (local address 127.0.0.1:44454)
0929 10:58:32.340885 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:32.340894 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 10:58:32.340929 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:58:32.341297 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:58:32.341298 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:58:32.341300 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:58:32.341334 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:32.341337 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:32.342672 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:32.342673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:32.343648 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:32.343651 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:32.344682 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:32.344684 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:32.344748 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:32.344756 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 10:58:32.344821 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 10:58:32.344853 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":43,"thread_start_us":84,"threads_started":1}
I20260929 10:58:32.345369 29735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:32.340495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52298 (local address 127.10.238.193:37817)
0929 10:58:32.340685 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:32.340690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:32.340796 (+ 106us) server_negotiation.cc:407] Connection header received
0929 10:58:32.340863 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:32.340866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:32.340886 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:32.341238 (+ 352us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:58:32.341360 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:32.342545 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:32.343862 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:32.344596 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:32.345011 (+ 415us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:32.345157 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:32.345193 (+ 36us) server_negotiation.cc:299] Negotiation successful
0929 10:58:32.345239 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":85,"threads_started":1}
I20260929 10:58:32.345578 29521 tablet_service.cc:1467] Processing CreateTablet for tablet 508320fb24cb4a9cbd28bbc490be3f17 (DEFAULT_TABLE table=test-table [id=cab35ebca11a4cd2860187daeb7c76af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:58:32.345939 29521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 508320fb24cb4a9cbd28bbc490be3f17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:58:32.345916 29655 tablet_service.cc:1467] Processing CreateTablet for tablet 508320fb24cb4a9cbd28bbc490be3f17 (DEFAULT_TABLE table=test-table [id=cab35ebca11a4cd2860187daeb7c76af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:58:32.346174 29655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 508320fb24cb4a9cbd28bbc490be3f17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:58:32.348359 29738 tablet_bootstrap.cc:492] T 508320fb24cb4a9cbd28bbc490be3f17 P c31719669d5343d582466e84031d62ea: Bootstrap starting.
I20260929 10:58:32.348599 29739 tablet_bootstrap.cc:492] T 508320fb24cb4a9cbd28bbc490be3f17 P 483d50e68119450fab798f37ebcdead0: Bootstrap starting.
I20260929 10:58:32.349061 29738 tablet_bootstrap.cc:654] T 508320fb24cb4a9cbd28bbc490be3f17 P c31719669d5343d582466e84031d62ea: Neither blocks nor log segments found. Creating new log.
I20260929 10:58:32.349097 29387 tablet_service.cc:1467] Processing CreateTablet for tablet 508320fb24cb4a9cbd28bbc490be3f17 (DEFAULT_TABLE table=test-table [id=cab35ebca11a4cd2860187daeb7c76af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:58:32.349324 29739 tablet_bootstrap.cc:654] T 508320fb24cb4a9cbd28bbc490be3f17 P 483d50e68119450fab798f37ebcdead0: Neither blocks nor log segments found. Creating new log.
I20260929 10:58:32.349470 29387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 508320fb24cb4a9cbd28bbc490be3f17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:58:32.349713 29738 log.cc:824] T 508320fb24cb4a9cbd28bbc490be3f17 P c31719669d5343d582466e84031d62ea: Log is configured to *not* fsync() on all Append() calls
I20260929 10:58:32.349927 29739 log.cc:824] T 508320fb24cb4a9cbd28bbc490be3f17 P 483d50e68119450fab798f37ebcdead0: Log is configured to *not* fsync() on all Append() calls
I20260929 10:58:32.350454 29738 tablet_bootstrap.cc:492] T 508320fb24cb4a9cbd28bbc490be3f17 P c31719669d5343d582466e84031d62ea: No bootstrap required, opened a new log
I20260929 10:58:32.350538 29738 ts_tablet_manager.cc:1397] T 508320fb24cb4a9cbd28bbc490be3f17 P c31719669d5343d582466e84031d62ea: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 10:58:32.352058 29738 raft_consensus.cc:348] T 508320fb24cb4a9cbd28bbc490be3f17 P c31719669d5343d582466e84031d62ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0706d043bb148f9b532fcdd8c52ce2a" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 37817 } } peers { permanent_uuid: "483d50e68119450fab798f37ebcdead0" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 43107 } } peers { permanent_uuid: "c31719669d5343d582466e84031d62ea" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35333 } }
I20260929 10:58:32.352196 29738 raft_consensus.cc:374] T 508320fb24cb4a9cbd28bbc490be3f17 P c31719669d5343d582466e84031d62ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:58:32.352218 29738 raft_consensus.cc:729] T 508320fb24cb4a9cbd28bbc490be3f17 P c31719669d5343d582466e84031d62ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c31719669d5343d582466e84031d62ea, State: Initialized, Role: FOLLOWER
I20260929 10:58:32.352339 29738 consensus_queue.cc:260] T 508320fb24cb4a9cbd28bbc490be3f17 P c31719669d5343d582466e84031d62ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0706d043bb148f9b532fcdd8c52ce2a" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 37817 } } peers { permanent_uuid: "483d50e68119450fab798f37ebcdead0" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 43107 } } peers { permanent_uuid: "c31719669d5343d582466e84031d62ea" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35333 } }
I20260929 10:58:32.352659 29738 ts_tablet_manager.cc:1428] T 508320fb24cb4a9cbd28bbc490be3f17 P c31719669d5343d582466e84031d62ea: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 10:58:32.352702 29587 heartbeater.cc:503] Master 127.10.238.254:45063 was elected leader, sending a full tablet report...
I20260929 10:58:32.352771 29743 tablet_bootstrap.cc:492] T 508320fb24cb4a9cbd28bbc490be3f17 P a0706d043bb148f9b532fcdd8c52ce2a: Bootstrap starting.
I20260929 10:58:32.353317 29739 tablet_bootstrap.cc:492] T 508320fb24cb4a9cbd28bbc490be3f17 P 483d50e68119450fab798f37ebcdead0: No bootstrap required, opened a new log
I20260929 10:58:32.353391 29739 ts_tablet_manager.cc:1397] T 508320fb24cb4a9cbd28bbc490be3f17 P 483d50e68119450fab798f37ebcdead0: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260929 10:58:32.353446 29743 tablet_bootstrap.cc:654] T 508320fb24cb4a9cbd28bbc490be3f17 P a0706d043bb148f9b532fcdd8c52ce2a: Neither blocks nor log segments found. Creating new log.
I20260929 10:58:32.354013 29743 log.cc:824] T 508320fb24cb4a9cbd28bbc490be3f17 P a0706d043bb148f9b532fcdd8c52ce2a: Log is configured to *not* fsync() on all Append() calls
I20260929 10:58:32.354781 29743 tablet_bootstrap.cc:492] T 508320fb24cb4a9cbd28bbc490be3f17 P a0706d043bb148f9b532fcdd8c52ce2a: No bootstrap required, opened a new log
I20260929 10:58:32.354851 29743 ts_tablet_manager.cc:1397] T 508320fb24cb4a9cbd28bbc490be3f17 P a0706d043bb148f9b532fcdd8c52ce2a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 10:58:32.355530 29739 raft_consensus.cc:348] T 508320fb24cb4a9cbd28bbc490be3f17 P 483d50e68119450fab798f37ebcdead0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0706d043bb148f9b532fcdd8c52ce2a" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 37817 } } peers { permanent_uuid: "483d50e68119450fab798f37ebcdead0" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 43107 } } peers { permanent_uuid: "c31719669d5343d582466e84031d62ea" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35333 } }
I20260929 10:58:32.355727 29739 raft_consensus.cc:374] T 508320fb24cb4a9cbd28bbc490be3f17 P 483d50e68119450fab798f37ebcdead0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:58:32.355775 29739 raft_consensus.cc:729] T 508320fb24cb4a9cbd28bbc490be3f17 P 483d50e68119450fab798f37ebcdead0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 483d50e68119450fab798f37ebcdead0, State: Initialized, Role: FOLLOWER
I20260929 10:58:32.355885 29739 consensus_queue.cc:260] T 508320fb24cb4a9cbd28bbc490be3f17 P 483d50e68119450fab798f37ebcdead0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0706d043bb148f9b532fcdd8c52ce2a" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 37817 } } peers { permanent_uuid: "483d50e68119450fab798f37ebcdead0" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 43107 } } peers { permanent_uuid: "c31719669d5343d582466e84031d62ea" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35333 } }
I20260929 10:58:32.356169 29739 ts_tablet_manager.cc:1428] T 508320fb24cb4a9cbd28bbc490be3f17 P 483d50e68119450fab798f37ebcdead0: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260929 10:58:32.356218 29721 heartbeater.cc:503] Master 127.10.238.254:45063 was elected leader, sending a full tablet report...
I20260929 10:58:32.356740 29743 raft_consensus.cc:348] T 508320fb24cb4a9cbd28bbc490be3f17 P a0706d043bb148f9b532fcdd8c52ce2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0706d043bb148f9b532fcdd8c52ce2a" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 37817 } } peers { permanent_uuid: "483d50e68119450fab798f37ebcdead0" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 43107 } } peers { permanent_uuid: "c31719669d5343d582466e84031d62ea" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35333 } }
I20260929 10:58:32.356871 29743 raft_consensus.cc:374] T 508320fb24cb4a9cbd28bbc490be3f17 P a0706d043bb148f9b532fcdd8c52ce2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:58:32.356892 29743 raft_consensus.cc:729] T 508320fb24cb4a9cbd28bbc490be3f17 P a0706d043bb148f9b532fcdd8c52ce2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0706d043bb148f9b532fcdd8c52ce2a, State: Initialized, Role: FOLLOWER
I20260929 10:58:32.357013 29743 consensus_queue.cc:260] T 508320fb24cb4a9cbd28bbc490be3f17 P a0706d043bb148f9b532fcdd8c52ce2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0706d043bb148f9b532fcdd8c52ce2a" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 37817 } } peers { permanent_uuid: "483d50e68119450fab798f37ebcdead0" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 43107 } } peers { permanent_uuid: "c31719669d5343d582466e84031d62ea" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35333 } }
I20260929 10:58:32.357316 29743 ts_tablet_manager.cc:1428] T 508320fb24cb4a9cbd28bbc490be3f17 P a0706d043bb148f9b532fcdd8c52ce2a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 10:58:32.357362 29453 heartbeater.cc:503] Master 127.10.238.254:45063 was elected leader, sending a full tablet report...
W20260929 10:58:32.401809 29588 tablet.cc:2369] T 508320fb24cb4a9cbd28bbc490be3f17 P c31719669d5343d582466e84031d62ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 10:58:32.491567 29454 tablet.cc:2369] T 508320fb24cb4a9cbd28bbc490be3f17 P a0706d043bb148f9b532fcdd8c52ce2a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:58:32.512559 29742 raft_consensus.cc:482] T 508320fb24cb4a9cbd28bbc490be3f17 P c31719669d5343d582466e84031d62ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:58:32.512681 29742 raft_consensus.cc:504] T 508320fb24cb4a9cbd28bbc490be3f17 P c31719669d5343d582466e84031d62ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0706d043bb148f9b532fcdd8c52ce2a" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 37817 } } peers { permanent_uuid: "483d50e68119450fab798f37ebcdead0" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 43107 } } peers { permanent_uuid: "c31719669d5343d582466e84031d62ea" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35333 } }
I20260929 10:58:32.513082 29742 leader_election.cc:288] T 508320fb24cb4a9cbd28bbc490be3f17 P c31719669d5343d582466e84031d62ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0706d043bb148f9b532fcdd8c52ce2a (127.10.238.193:37817), 483d50e68119450fab798f37ebcdead0 (127.10.238.195:43107)
I20260929 10:58:32.516011 29589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:32.513249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:37817 (local address 127.10.238.194:50483)
0929 10:58:32.513332 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:32.513344 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:58:32.513392 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:58:32.513580 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:58:32.513581 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:58:32.513583 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:58:32.513661 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:32.513662 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:32.514415 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:32.514416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:32.515183 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:32.515185 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:32.515750 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:32.515752 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:32.515836 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:32.515839 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 10:58:32.515896 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 10:58:32.515928 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260929 10:58:32.516036 29735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:32.513374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:50483 (local address 127.10.238.193:37817)
0929 10:58:32.513437 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:32.513440 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:32.513452 (+ 12us) server_negotiation.cc:407] Connection header received
0929 10:58:32.513484 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:32.513486 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:32.513496 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:32.513575 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:58:32.513749 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:32.514332 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:32.515288 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:32.515661 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:32.515717 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:32.515725 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:32.515912 (+ 187us) server_negotiation.cc:299] Negotiation successful
0929 10:58:32.515960 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 10:58:32.516116 29747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:32.513412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.195:43107 (local address 127.10.238.194:47395)
0929 10:58:32.513579 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:32.513591 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:58:32.513640 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:58:32.513759 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:58:32.513760 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:58:32.513761 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:58:32.513830 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:32.513832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:32.514497 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:32.514497 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:32.515263 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:32.515266 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:32.515845 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:32.515846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:32.515933 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:32.515938 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 10:58:32.515994 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 10:58:32.516030 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":98,"threads_started":1}
I20260929 10:58:32.516137 29737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:32.513569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:47395 (local address 127.10.238.195:43107)
0929 10:58:32.513617 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:32.513619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:32.513629 (+ 10us) server_negotiation.cc:407] Connection header received
0929 10:58:32.513695 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:32.513696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:32.513706 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:32.513755 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:58:32.513924 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:32.514407 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:32.515353 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:32.515757 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:32.515831 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:32.515839 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:32.516014 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 10:58:32.516061 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260929 10:58:32.516363 29407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508320fb24cb4a9cbd28bbc490be3f17" candidate_uuid: "c31719669d5343d582466e84031d62ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0706d043bb148f9b532fcdd8c52ce2a" is_pre_election: true
I20260929 10:58:32.516516 29407 raft_consensus.cc:2454] T 508320fb24cb4a9cbd28bbc490be3f17 P a0706d043bb148f9b532fcdd8c52ce2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c31719669d5343d582466e84031d62ea in term 0.
I20260929 10:58:32.516686 29675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508320fb24cb4a9cbd28bbc490be3f17" candidate_uuid: "c31719669d5343d582466e84031d62ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "483d50e68119450fab798f37ebcdead0" is_pre_election: true
I20260929 10:58:32.516767 29475 leader_election.cc:302] T 508320fb24cb4a9cbd28bbc490be3f17 P c31719669d5343d582466e84031d62ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0706d043bb148f9b532fcdd8c52ce2a, c31719669d5343d582466e84031d62ea; no voters:
I20260929 10:58:32.516866 29675 raft_consensus.cc:2454] T 508320fb24cb4a9cbd28bbc490be3f17 P 483d50e68119450fab798f37ebcdead0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c31719669d5343d582466e84031d62ea in term 0.
I20260929 10:58:32.516898 29742 raft_consensus.cc:2781] T 508320fb24cb4a9cbd28bbc490be3f17 P c31719669d5343d582466e84031d62ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:58:32.516969 29742 raft_consensus.cc:482] T 508320fb24cb4a9cbd28bbc490be3f17 P c31719669d5343d582466e84031d62ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:58:32.517009 29742 raft_consensus.cc:3037] T 508320fb24cb4a9cbd28bbc490be3f17 P c31719669d5343d582466e84031d62ea [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:58:32.517812 29742 raft_consensus.cc:504] T 508320fb24cb4a9cbd28bbc490be3f17 P c31719669d5343d582466e84031d62ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0706d043bb148f9b532fcdd8c52ce2a" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 37817 } } peers { permanent_uuid: "483d50e68119450fab798f37ebcdead0" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 43107 } } peers { permanent_uuid: "c31719669d5343d582466e84031d62ea" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35333 } }
I20260929 10:58:32.517918 29742 leader_election.cc:288] T 508320fb24cb4a9cbd28bbc490be3f17 P c31719669d5343d582466e84031d62ea [CANDIDATE]: Term 1 election: Requested vote from peers a0706d043bb148f9b532fcdd8c52ce2a (127.10.238.193:37817), 483d50e68119450fab798f37ebcdead0 (127.10.238.195:43107)
I20260929 10:58:32.518162 29407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508320fb24cb4a9cbd28bbc490be3f17" candidate_uuid: "c31719669d5343d582466e84031d62ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0706d043bb148f9b532fcdd8c52ce2a"
I20260929 10:58:32.518200 29675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508320fb24cb4a9cbd28bbc490be3f17" candidate_uuid: "c31719669d5343d582466e84031d62ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "483d50e68119450fab798f37ebcdead0"
I20260929 10:58:32.518236 29407 raft_consensus.cc:3037] T 508320fb24cb4a9cbd28bbc490be3f17 P a0706d043bb148f9b532fcdd8c52ce2a [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:58:32.518321 29675 raft_consensus.cc:3037] T 508320fb24cb4a9cbd28bbc490be3f17 P 483d50e68119450fab798f37ebcdead0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:58:32.519025 29407 raft_consensus.cc:2454] T 508320fb24cb4a9cbd28bbc490be3f17 P a0706d043bb148f9b532fcdd8c52ce2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c31719669d5343d582466e84031d62ea in term 1.
I20260929 10:58:32.519207 29475 leader_election.cc:302] T 508320fb24cb4a9cbd28bbc490be3f17 P c31719669d5343d582466e84031d62ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0706d043bb148f9b532fcdd8c52ce2a, c31719669d5343d582466e84031d62ea; no voters:
I20260929 10:58:32.519235 29675 raft_consensus.cc:2454] T 508320fb24cb4a9cbd28bbc490be3f17 P 483d50e68119450fab798f37ebcdead0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c31719669d5343d582466e84031d62ea in term 1.
I20260929 10:58:32.519306 29742 raft_consensus.cc:2781] T 508320fb24cb4a9cbd28bbc490be3f17 P c31719669d5343d582466e84031d62ea [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:58:32.519486 29742 raft_consensus.cc:686] T 508320fb24cb4a9cbd28bbc490be3f17 P c31719669d5343d582466e84031d62ea [term 1 LEADER]: Becoming Leader. State: Replica: c31719669d5343d582466e84031d62ea, State: Running, Role: LEADER
I20260929 10:58:32.519588 29742 consensus_queue.cc:237] T 508320fb24cb4a9cbd28bbc490be3f17 P c31719669d5343d582466e84031d62ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0706d043bb148f9b532fcdd8c52ce2a" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 37817 } } peers { permanent_uuid: "483d50e68119450fab798f37ebcdead0" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 43107 } } peers { permanent_uuid: "c31719669d5343d582466e84031d62ea" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35333 } }
I20260929 10:58:32.520489 29262 catalog_manager.cc:5495] T 508320fb24cb4a9cbd28bbc490be3f17 P c31719669d5343d582466e84031d62ea reported cstate change: term changed from 0 to 1, leader changed from <none> to c31719669d5343d582466e84031d62ea (127.10.238.194). New cstate: current_term: 1 leader_uuid: "c31719669d5343d582466e84031d62ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0706d043bb148f9b532fcdd8c52ce2a" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 37817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "483d50e68119450fab798f37ebcdead0" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 43107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c31719669d5343d582466e84031d62ea" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35333 } health_report { overall_health: HEALTHY } } }
I20260929 10:58:32.532730 29734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:32.530524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34228 (local address 127.10.238.194:35333)
0929 10:58:32.530579 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:32.530581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:32.530589 (+ 8us) server_negotiation.cc:407] Connection header received
0929 10:58:32.530656 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:32.530657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:32.530673 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:32.530737 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:58:32.530865 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:32.531316 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:32.532069 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:32.532263 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:32.532327 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:32.532442 (+ 115us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:58:32.532526 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:58:32.532602 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:32.532667 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 10:58:32.532676 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 10:58:32.534482 29675 raft_consensus.cc:1261] T 508320fb24cb4a9cbd28bbc490be3f17 P 483d50e68119450fab798f37ebcdead0 [term 1 FOLLOWER]: Refusing update from remote peer c31719669d5343d582466e84031d62ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:58:32.534502 29407 raft_consensus.cc:1261] T 508320fb24cb4a9cbd28bbc490be3f17 P a0706d043bb148f9b532fcdd8c52ce2a [term 1 FOLLOWER]: Refusing update from remote peer c31719669d5343d582466e84031d62ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:58:32.534832 29742 consensus_queue.cc:1035] T 508320fb24cb4a9cbd28bbc490be3f17 P c31719669d5343d582466e84031d62ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0706d043bb148f9b532fcdd8c52ce2a" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 37817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:58:32.534981 29748 consensus_queue.cc:1035] T 508320fb24cb4a9cbd28bbc490be3f17 P c31719669d5343d582466e84031d62ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "483d50e68119450fab798f37ebcdead0" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 43107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:58:32.537643 29752 mvcc.cc:205] Tried to move back new op lower bound from 7334623283338747904 to 7334623283280891904. Current Snapshot: MvccSnapshot[applied={T|T < 7334623283338747904}]
I20260929 10:58:32.538152 29753 mvcc.cc:205] Tried to move back new op lower bound from 7334623283338747904 to 7334623283280891904. Current Snapshot: MvccSnapshot[applied={T|T < 7334623283338747904 or (T in {7334623283338747904})}]
I20260929 10:58:32.539597 29262 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:58:32.539740 29262 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:58:32.540445 29262 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7b8fc327412c412a98bc589b3d13db19: Sending DeleteTablet for 3 replicas of tablet 508320fb24cb4a9cbd28bbc490be3f17
I20260929 10:58:32.540758 29387 tablet_service.cc:1514] Processing DeleteTablet for tablet 508320fb24cb4a9cbd28bbc490be3f17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:58:32 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:52298
I20260929 10:58:32.540809 29655 tablet_service.cc:1514] Processing DeleteTablet for tablet 508320fb24cb4a9cbd28bbc490be3f17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:58:32 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:44454
I20260929 10:58:32.540959 29521 tablet_service.cc:1514] Processing DeleteTablet for tablet 508320fb24cb4a9cbd28bbc490be3f17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:58:32 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:34214
I20260929 10:58:32.541074 29762 tablet_replica.cc:330] T 508320fb24cb4a9cbd28bbc490be3f17 P 483d50e68119450fab798f37ebcdead0: stopping tablet replica
I20260929 10:58:32.541162 29761 tablet_replica.cc:330] T 508320fb24cb4a9cbd28bbc490be3f17 P a0706d043bb148f9b532fcdd8c52ce2a: stopping tablet replica
I20260929 10:58:32.541285 29761 raft_consensus.cc:2229] T 508320fb24cb4a9cbd28bbc490be3f17 P a0706d043bb148f9b532fcdd8c52ce2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:58:32.541203 29762 raft_consensus.cc:2229] T 508320fb24cb4a9cbd28bbc490be3f17 P 483d50e68119450fab798f37ebcdead0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:58:32.541330 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 29324
I20260929 10:58:32.541396 29761 raft_consensus.cc:2258] T 508320fb24cb4a9cbd28bbc490be3f17 P a0706d043bb148f9b532fcdd8c52ce2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:58:32.541518 29762 raft_consensus.cc:2258] T 508320fb24cb4a9cbd28bbc490be3f17 P 483d50e68119450fab798f37ebcdead0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:58:32.542255 29762 ts_tablet_manager.cc:1905] T 508320fb24cb4a9cbd28bbc490be3f17 P 483d50e68119450fab798f37ebcdead0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:58:32.544200 29762 ts_tablet_manager.cc:1918] T 508320fb24cb4a9cbd28bbc490be3f17 P 483d50e68119450fab798f37ebcdead0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:58:32.544282 29762 log.cc:1196] T 508320fb24cb4a9cbd28bbc490be3f17 P 483d50e68119450fab798f37ebcdead0: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/ts-2/wal/wals/508320fb24cb4a9cbd28bbc490be3f17
I20260929 10:58:32.544556 29762 ts_tablet_manager.cc:1939] T 508320fb24cb4a9cbd28bbc490be3f17 P 483d50e68119450fab798f37ebcdead0: Deleting consensus metadata
I20260929 10:58:32.544983 29247 catalog_manager.cc:4841] TS 483d50e68119450fab798f37ebcdead0 (127.10.238.195:43107): tablet 508320fb24cb4a9cbd28bbc490be3f17 (table test-table [id=cab35ebca11a4cd2860187daeb7c76af]) successfully deleted
I20260929 10:58:32.545387 29763 tablet_replica.cc:330] T 508320fb24cb4a9cbd28bbc490be3f17 P c31719669d5343d582466e84031d62ea: stopping tablet replica
I20260929 10:58:32.545488 29763 raft_consensus.cc:2229] T 508320fb24cb4a9cbd28bbc490be3f17 P c31719669d5343d582466e84031d62ea [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:58:32.545578 29763 raft_consensus.cc:2258] T 508320fb24cb4a9cbd28bbc490be3f17 P c31719669d5343d582466e84031d62ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:58:32.546010 29763 ts_tablet_manager.cc:1905] T 508320fb24cb4a9cbd28bbc490be3f17 P c31719669d5343d582466e84031d62ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:58:32.547578 29763 ts_tablet_manager.cc:1918] T 508320fb24cb4a9cbd28bbc490be3f17 P c31719669d5343d582466e84031d62ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:58:32.547665 29763 log.cc:1196] T 508320fb24cb4a9cbd28bbc490be3f17 P c31719669d5343d582466e84031d62ea: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790679274510869-11195-0/minicluster-data/ts-1/wal/wals/508320fb24cb4a9cbd28bbc490be3f17
I20260929 10:58:32.547935 29763 ts_tablet_manager.cc:1939] T 508320fb24cb4a9cbd28bbc490be3f17 P c31719669d5343d582466e84031d62ea: Deleting consensus metadata
W20260929 10:58:32.548260 29247 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.238.193:37817 (error 108)
I20260929 10:58:32.548530 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 29458
I20260929 10:58:32.548523 29246 catalog_manager.cc:4841] TS c31719669d5343d582466e84031d62ea (127.10.238.194:35333): tablet 508320fb24cb4a9cbd28bbc490be3f17 (table test-table [id=cab35ebca11a4cd2860187daeb7c76af]) successfully deleted
I20260929 10:58:32.549149 29736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:32.548676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:37817 (local address 127.0.0.1:52312)
0929 10:58:32.549077 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:32.549112 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.193:37817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380}
W20260929 10:58:32.549371 29247 catalog_manager.cc:4584] TS a0706d043bb148f9b532fcdd8c52ce2a (127.10.238.193:37817): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 508320fb24cb4a9cbd28bbc490be3f17: Network error: Client connection negotiation failed: client connection to 127.10.238.193:37817: connect: Connection refused (error 111)
I20260929 10:58:32.553354 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 29592
I20260929 10:58:32.558715 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 29230
2026-09-29T10:58:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2094 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T10:58:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:58:32Z Disabled control of system clock
I20260929 10:58:32.586732 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:37233
--webserver_interface=127.10.238.254
--webserver_port=0
--builtin_ntp_servers=127.10.238.212:40251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:37233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 10:58:32.666591 29768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:58:32.666806 29768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:58:32.666826 29768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:58:32.667981 29768 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:58:32.668023 29768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:58:32.668036 29768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:58:32.668085 29768 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:58:32.668111 29768 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:58:32.669459 29768 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:40251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:37233
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:37233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:58:32.669708 29768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:58:32.669996 29768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:58:32.674896 29773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:32.674906 29774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:32.675007 29768 server_base.cc:1034] running on GCE node
W20260929 10:58:32.674906 29776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:32.675333 29768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:58:32.675563 29768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:58:32.676694 29768 hybrid_clock.cc:648] HybridClock initialized: now 1790679512676680 us; error 31 us; skew 500 ppm
I20260929 10:58:32.683604 29768 webserver.cc:460] Webserver started at http://127.10.238.254:39941/ using document root <none> and password file <none>
I20260929 10:58:32.683807 29768 fs_manager.cc:359] Metadata directory not provided
I20260929 10:58:32.683848 29768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:58:32.683935 29768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:58:32.684900 29768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/master-0/data/instance:
uuid: "526ccc0ae8f145659bee9181f2cd9825"
format_stamp: "Formatted at 2026-09-29 10:58:32 on dist-test-slave-8nrt"
server_key: "79f87250768e189c19f13a898bf1ab2c"
server_key_iv: "65e0c7e50879ad0dc65f7e7531add91f"
server_key_version: "encryptionkey@0"
I20260929 10:58:32.685185 29768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/master-0/wal/instance:
uuid: "526ccc0ae8f145659bee9181f2cd9825"
format_stamp: "Formatted at 2026-09-29 10:58:32 on dist-test-slave-8nrt"
server_key: "79f87250768e189c19f13a898bf1ab2c"
server_key_iv: "65e0c7e50879ad0dc65f7e7531add91f"
server_key_version: "encryptionkey@0"
I20260929 10:58:32.686450 29768 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 10:58:32.687134 29782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:32.687316 29768 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 10:58:32.687402 29768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "526ccc0ae8f145659bee9181f2cd9825"
format_stamp: "Formatted at 2026-09-29 10:58:32 on dist-test-slave-8nrt"
server_key: "79f87250768e189c19f13a898bf1ab2c"
server_key_iv: "65e0c7e50879ad0dc65f7e7531add91f"
server_key_version: "encryptionkey@0"
I20260929 10:58:32.687455 29768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:58:32.697325 29768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:58:32.697610 29768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:58:32.697757 29768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:58:32.701670 29768 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.254:37233
I20260929 10:58:32.701727 29834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.254:37233 every 8 connection(s)
I20260929 10:58:32.702061 29768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:58:32.703024 29835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:58:32.705277 29835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 526ccc0ae8f145659bee9181f2cd9825: Bootstrap starting.
I20260929 10:58:32.705853 29835 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 526ccc0ae8f145659bee9181f2cd9825: Neither blocks nor log segments found. Creating new log.
I20260929 10:58:32.706151 29835 log.cc:824] T 00000000000000000000000000000000 P 526ccc0ae8f145659bee9181f2cd9825: Log is configured to *not* fsync() on all Append() calls
I20260929 10:58:32.706867 29835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 526ccc0ae8f145659bee9181f2cd9825: No bootstrap required, opened a new log
I20260929 10:58:32.708472 29835 raft_consensus.cc:348] T 00000000000000000000000000000000 P 526ccc0ae8f145659bee9181f2cd9825 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "526ccc0ae8f145659bee9181f2cd9825" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 37233 } }
I20260929 10:58:32.708603 29835 raft_consensus.cc:374] T 00000000000000000000000000000000 P 526ccc0ae8f145659bee9181f2cd9825 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:58:32.708626 29835 raft_consensus.cc:729] T 00000000000000000000000000000000 P 526ccc0ae8f145659bee9181f2cd9825 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 526ccc0ae8f145659bee9181f2cd9825, State: Initialized, Role: FOLLOWER
I20260929 10:58:32.708804 29835 consensus_queue.cc:260] T 00000000000000000000000000000000 P 526ccc0ae8f145659bee9181f2cd9825 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "526ccc0ae8f145659bee9181f2cd9825" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 37233 } }
I20260929 10:58:32.708879 29835 raft_consensus.cc:388] T 00000000000000000000000000000000 P 526ccc0ae8f145659bee9181f2cd9825 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:58:32.708925 29835 raft_consensus.cc:482] T 00000000000000000000000000000000 P 526ccc0ae8f145659bee9181f2cd9825 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:58:32.708976 29835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 526ccc0ae8f145659bee9181f2cd9825 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:58:32.709581 29835 raft_consensus.cc:504] T 00000000000000000000000000000000 P 526ccc0ae8f145659bee9181f2cd9825 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "526ccc0ae8f145659bee9181f2cd9825" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 37233 } }
I20260929 10:58:32.709693 29835 leader_election.cc:302] T 00000000000000000000000000000000 P 526ccc0ae8f145659bee9181f2cd9825 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 526ccc0ae8f145659bee9181f2cd9825; no voters:
I20260929 10:58:32.709875 29835 leader_election.cc:288] T 00000000000000000000000000000000 P 526ccc0ae8f145659bee9181f2cd9825 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:58:32.710021 29838 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 526ccc0ae8f145659bee9181f2cd9825 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:58:32.710135 29835 sys_catalog.cc:564] T 00000000000000000000000000000000 P 526ccc0ae8f145659bee9181f2cd9825 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:58:32.710207 29838 raft_consensus.cc:686] T 00000000000000000000000000000000 P 526ccc0ae8f145659bee9181f2cd9825 [term 1 LEADER]: Becoming Leader. State: Replica: 526ccc0ae8f145659bee9181f2cd9825, State: Running, Role: LEADER
I20260929 10:58:32.710316 29838 consensus_queue.cc:237] T 00000000000000000000000000000000 P 526ccc0ae8f145659bee9181f2cd9825 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "526ccc0ae8f145659bee9181f2cd9825" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 37233 } }
I20260929 10:58:32.710678 29840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 526ccc0ae8f145659bee9181f2cd9825 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 526ccc0ae8f145659bee9181f2cd9825. Latest consensus state: current_term: 1 leader_uuid: "526ccc0ae8f145659bee9181f2cd9825" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "526ccc0ae8f145659bee9181f2cd9825" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 37233 } } }
I20260929 10:58:32.710765 29839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 526ccc0ae8f145659bee9181f2cd9825 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "526ccc0ae8f145659bee9181f2cd9825" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "526ccc0ae8f145659bee9181f2cd9825" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 37233 } } }
I20260929 10:58:32.710784 29840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 526ccc0ae8f145659bee9181f2cd9825 [sys.catalog]: This master's current role is: LEADER
I20260929 10:58:32.710886 29839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 526ccc0ae8f145659bee9181f2cd9825 [sys.catalog]: This master's current role is: LEADER
I20260929 10:58:32.710956 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 29768
I20260929 10:58:32.711052 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/master-0/wal/instance
I20260929 10:58:32.711287 11195 external_mini_cluster.cc:1428] Setting key 53d2587a5ca432b633db10a3a1db8106
W20260929 10:58:32.712798 29856 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 526ccc0ae8f145659bee9181f2cd9825: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:58:32.712853 29856 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:58:32.712913 29845 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:58:32.713061 29845 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:58:32.714576 29845 catalog_manager.cc:1347] Generated new cluster ID: b80c0457dcc04f59a33335949aaddbbf
I20260929 10:58:32.714629 29845 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:58:32.718060 29853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:32.711971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48352 (local address 127.10.238.254:37233)
0929 10:58:32.712610 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:32.712619 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:32.712656 (+ 37us) server_negotiation.cc:407] Connection header received
0929 10:58:32.712770 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:32.712776 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:32.712837 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:32.712984 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:32.713348 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:32.714320 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:32.715556 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:32.716212 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:32.716689 (+ 477us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:32.717083 (+ 394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:32.717087 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:32.717088 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:58:32.717331 (+ 243us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:32.717374 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:58:32.717390 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:32.717710 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:32.717754 (+ 44us) server_negotiation.cc:299] Negotiation successful
0929 10:58:32.717836 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":89,"threads_started":1}
I20260929 10:58:32.738611 29845 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:58:32.739348 29845 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:58:32.747304 29845 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 526ccc0ae8f145659bee9181f2cd9825: Generated new TSK 0
I20260929 10:58:32.747522 29845 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:58:32.754266 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.193:0
--local_ip_for_outbound_sockets=127.10.238.193
--webserver_interface=127.10.238.193
--webserver_port=0
--tserver_master_addrs=127.10.238.254:37233
--builtin_ntp_servers=127.10.238.212:40251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 10:58:32.854061 29859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:58:32.854291 29859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:58:32.854327 29859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:58:32.855472 29859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:58:32.855542 29859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:58:32.855588 29859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.193
I20260929 10:58:32.856958 29859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:40251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.238.193
--webserver_port=0
--tserver_master_addrs=127.10.238.254:37233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.193
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:58:32.857221 29859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:58:32.857546 29859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:58:32.863333 29865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:32.863415 29864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:32.863463 29859 server_base.cc:1034] running on GCE node
W20260929 10:58:32.863363 29867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:32.863792 29859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:58:32.864099 29859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:58:32.865281 29859 hybrid_clock.cc:648] HybridClock initialized: now 1790679512865234 us; error 33 us; skew 500 ppm
I20260929 10:58:32.866607 29859 webserver.cc:460] Webserver started at http://127.10.238.193:38639/ using document root <none> and password file <none>
I20260929 10:58:32.866876 29859 fs_manager.cc:359] Metadata directory not provided
I20260929 10:58:32.866950 29859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:58:32.867089 29859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:58:32.868135 29859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/ts-0/data/instance:
uuid: "216d2fe7069f4712b0a28d358f3d187b"
format_stamp: "Formatted at 2026-09-29 10:58:32 on dist-test-slave-8nrt"
server_key: "a995fd7b2a3d8f3c535e116c706ad67b"
server_key_iv: "53287c1c4c9bcccf141595cb276f016d"
server_key_version: "encryptionkey@0"
I20260929 10:58:32.868508 29859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/ts-0/wal/instance:
uuid: "216d2fe7069f4712b0a28d358f3d187b"
format_stamp: "Formatted at 2026-09-29 10:58:32 on dist-test-slave-8nrt"
server_key: "a995fd7b2a3d8f3c535e116c706ad67b"
server_key_iv: "53287c1c4c9bcccf141595cb276f016d"
server_key_version: "encryptionkey@0"
I20260929 10:58:32.869890 29859 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 10:58:32.870577 29873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:32.870743 29859 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:58:32.870838 29859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/ts-0/wal
uuid: "216d2fe7069f4712b0a28d358f3d187b"
format_stamp: "Formatted at 2026-09-29 10:58:32 on dist-test-slave-8nrt"
server_key: "a995fd7b2a3d8f3c535e116c706ad67b"
server_key_iv: "53287c1c4c9bcccf141595cb276f016d"
server_key_version: "encryptionkey@0"
I20260929 10:58:32.870919 29859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:58:32.882530 29859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:58:32.882819 29859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:58:32.882967 29859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:58:32.883167 29859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:58:32.883484 29859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:58:32.883540 29859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:32.883580 29859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:58:32.883610 29859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:32.889401 29859 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.193:35623
I20260929 10:58:32.889446 29986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.193:35623 every 8 connection(s)
I20260929 10:58:32.889781 29859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
I20260929 10:58:32.892611 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 29859
I20260929 10:58:32.892769 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/ts-0/wal/instance
I20260929 10:58:32.892983 11195 external_mini_cluster.cc:1428] Setting key 83bfd7510017a51679743b465a40fc51
I20260929 10:58:32.893962 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.194:0
--local_ip_for_outbound_sockets=127.10.238.194
--webserver_interface=127.10.238.194
--webserver_port=0
--tserver_master_addrs=127.10.238.254:37233
--builtin_ntp_servers=127.10.238.212:40251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 10:58:32.895668 29853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:32.890576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:60851 (local address 127.10.238.254:37233)
0929 10:58:32.890620 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:32.890622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:32.890873 (+ 251us) server_negotiation.cc:407] Connection header received
0929 10:58:32.891410 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:32.891412 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:32.891423 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:32.891475 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:32.892096 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:32.892587 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:32.894058 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:32.894334 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:32.894392 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:32.895338 (+ 946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:32.895339 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:32.895340 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:58:32.895345 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:32.895363 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:58:32.895368 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:32.895429 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:32.895531 (+ 102us) server_negotiation.cc:299] Negotiation successful
0929 10:58:32.895564 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 10:58:32.895766 29989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:32.890567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:37233 (local address 127.10.238.193:60851)
0929 10:58:32.890747 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:32.890779 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 10:58:32.891318 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:58:32.891525 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:58:32.891528 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:58:32.891536 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:58:32.892025 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:32.892029 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:32.892687 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:32.892689 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:32.893951 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:32.893955 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:32.894433 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:32.894434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:32.894595 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:32.895144 (+ 549us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:58:32.895145 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:58:32.895187 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:58:32.895217 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:58:32.895445 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:58:32.895450 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:58:32.895452 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 10:58:32.895506 (+ 54us) client_negotiation.cc:240] Negotiation successful
0929 10:58:32.895584 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
I20260929 10:58:32.896232 29987 heartbeater.cc:347] Connected to a master server at 127.10.238.254:37233
I20260929 10:58:32.896368 29987 heartbeater.cc:464] Registering TS with master...
I20260929 10:58:32.896575 29987 heartbeater.cc:511] Master 127.10.238.254:37233 requested a full tablet report, sending...
I20260929 10:58:32.897111 29799 ts_manager.cc:194] Registered new tserver with Master: 216d2fe7069f4712b0a28d358f3d187b (127.10.238.193:35623)
I20260929 10:58:32.897746 29799 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.238.193:60851
W20260929 10:58:32.976653 29990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:58:32.976886 29990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:58:32.976923 29990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:58:32.978096 29990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:58:32.978158 29990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:58:32.978201 29990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.194
I20260929 10:58:32.979615 29990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:40251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.238.194
--webserver_port=0
--tserver_master_addrs=127.10.238.254:37233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.194
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:58:32.979866 29990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:58:32.980151 29990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:58:32.985118 29996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:32.985096 29998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:32.985281 29990 server_base.cc:1034] running on GCE node
W20260929 10:58:32.985328 29995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:32.985524 29990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:58:32.985774 29990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:58:32.986939 29990 hybrid_clock.cc:648] HybridClock initialized: now 1790679512986935 us; error 43 us; skew 500 ppm
I20260929 10:58:32.988178 29990 webserver.cc:460] Webserver started at http://127.10.238.194:33495/ using document root <none> and password file <none>
I20260929 10:58:32.988417 29990 fs_manager.cc:359] Metadata directory not provided
I20260929 10:58:32.988489 29990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:58:32.988624 29990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:58:32.989707 29990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/ts-1/data/instance:
uuid: "ad2af08371944acba50413b03ebf9ed4"
format_stamp: "Formatted at 2026-09-29 10:58:32 on dist-test-slave-8nrt"
server_key: "60ce795d8cdb819751e2e4a39e2cc173"
server_key_iv: "6ff1d7987623275910b8138ee3f9b27a"
server_key_version: "encryptionkey@0"
I20260929 10:58:32.990084 29990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/ts-1/wal/instance:
uuid: "ad2af08371944acba50413b03ebf9ed4"
format_stamp: "Formatted at 2026-09-29 10:58:32 on dist-test-slave-8nrt"
server_key: "60ce795d8cdb819751e2e4a39e2cc173"
server_key_iv: "6ff1d7987623275910b8138ee3f9b27a"
server_key_version: "encryptionkey@0"
I20260929 10:58:32.991434 29990 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 10:58:32.992128 30004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:32.992287 29990 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:58:32.992383 29990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/ts-1/wal
uuid: "ad2af08371944acba50413b03ebf9ed4"
format_stamp: "Formatted at 2026-09-29 10:58:32 on dist-test-slave-8nrt"
server_key: "60ce795d8cdb819751e2e4a39e2cc173"
server_key_iv: "6ff1d7987623275910b8138ee3f9b27a"
server_key_version: "encryptionkey@0"
I20260929 10:58:32.992462 29990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:58:33.038633 29990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:58:33.039004 29990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:58:33.039180 29990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:58:33.039443 29990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:58:33.039803 29990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:58:33.039858 29990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:33.039893 29990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:58:33.039919 29990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:33.046093 29990 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.194:34429
I20260929 10:58:33.046139 30117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.194:34429 every 8 connection(s)
I20260929 10:58:33.046532 29990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
I20260929 10:58:33.049875 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 29990
I20260929 10:58:33.049998 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/ts-1/wal/instance
I20260929 10:58:33.050205 11195 external_mini_cluster.cc:1428] Setting key 4ae45377a6f1abbd7bc8ce89b406eb59
I20260929 10:58:33.051236 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.195:0
--local_ip_for_outbound_sockets=127.10.238.195
--webserver_interface=127.10.238.195
--webserver_port=0
--tserver_master_addrs=127.10.238.254:37233
--builtin_ntp_servers=127.10.238.212:40251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 10:58:33.052105 29853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:33.047296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:40303 (local address 127.10.238.254:37233)
0929 10:58:33.047378 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:33.047380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:33.047595 (+ 215us) server_negotiation.cc:407] Connection header received
0929 10:58:33.048064 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:33.048065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:33.048074 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:33.048169 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:33.048843 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:33.049409 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:33.050530 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:33.050809 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:33.050873 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:33.051796 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:33.051797 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:33.051798 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:58:33.051803 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:33.051820 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:58:33.051825 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:33.051888 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:33.051958 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 10:58:33.051989 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 10:58:33.052210 30120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:33.047282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:37233 (local address 127.10.238.194:40303)
0929 10:58:33.047467 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:33.047509 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 10:58:33.047980 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:58:33.048228 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:58:33.048231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:58:33.048238 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:58:33.048758 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:33.048762 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:33.049491 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:33.049491 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:33.050440 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:33.050443 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:33.050897 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:33.050898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:33.051045 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:33.051655 (+ 610us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:58:33.051655 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:58:33.051685 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:58:33.051708 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:58:33.051888 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:58:33.051890 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:58:33.051893 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 10:58:33.051938 (+ 45us) client_negotiation.cc:240] Negotiation successful
0929 10:58:33.052012 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
I20260929 10:58:33.053054 30118 heartbeater.cc:347] Connected to a master server at 127.10.238.254:37233
I20260929 10:58:33.053135 30118 heartbeater.cc:464] Registering TS with master...
I20260929 10:58:33.053371 30118 heartbeater.cc:511] Master 127.10.238.254:37233 requested a full tablet report, sending...
I20260929 10:58:33.053826 29799 ts_manager.cc:194] Registered new tserver with Master: ad2af08371944acba50413b03ebf9ed4 (127.10.238.194:34429)
I20260929 10:58:33.054245 29799 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.238.194:40303
W20260929 10:58:33.135426 30121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:58:33.135677 30121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:58:33.135715 30121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:58:33.136965 30121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:58:33.137027 30121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:58:33.137063 30121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.195
I20260929 10:58:33.138449 30121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:40251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.238.195
--webserver_port=0
--tserver_master_addrs=127.10.238.254:37233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.195
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:58:33.138722 30121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:58:33.139029 30121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:58:33.144227 30129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:33.144220 30127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:33.144222 30126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:33.144239 30121 server_base.cc:1034] running on GCE node
I20260929 10:58:33.144703 30121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:58:33.144945 30121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:58:33.146135 30121 hybrid_clock.cc:648] HybridClock initialized: now 1790679513146123 us; error 28 us; skew 500 ppm
I20260929 10:58:33.147475 30121 webserver.cc:460] Webserver started at http://127.10.238.195:33529/ using document root <none> and password file <none>
I20260929 10:58:33.147778 30121 fs_manager.cc:359] Metadata directory not provided
I20260929 10:58:33.147831 30121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:58:33.147992 30121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:58:33.149153 30121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/ts-2/data/instance:
uuid: "2cc64fe379fd497b8063ab4786988009"
format_stamp: "Formatted at 2026-09-29 10:58:33 on dist-test-slave-8nrt"
server_key: "fde6ff242cc068376201a0b3300816ca"
server_key_iv: "203b32e7dc44cd456013c6ac3016ed54"
server_key_version: "encryptionkey@0"
I20260929 10:58:33.149617 30121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/ts-2/wal/instance:
uuid: "2cc64fe379fd497b8063ab4786988009"
format_stamp: "Formatted at 2026-09-29 10:58:33 on dist-test-slave-8nrt"
server_key: "fde6ff242cc068376201a0b3300816ca"
server_key_iv: "203b32e7dc44cd456013c6ac3016ed54"
server_key_version: "encryptionkey@0"
I20260929 10:58:33.150983 30121 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:58:33.151762 30135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:33.151934 30121 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:58:33.152015 30121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/ts-2/wal
uuid: "2cc64fe379fd497b8063ab4786988009"
format_stamp: "Formatted at 2026-09-29 10:58:33 on dist-test-slave-8nrt"
server_key: "fde6ff242cc068376201a0b3300816ca"
server_key_iv: "203b32e7dc44cd456013c6ac3016ed54"
server_key_version: "encryptionkey@0"
I20260929 10:58:33.152069 30121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:58:33.185688 30121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:58:33.185981 30121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:58:33.186080 30121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:58:33.186276 30121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:58:33.186636 30121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:58:33.186674 30121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:33.186743 30121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:58:33.186770 30121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:33.192792 30121 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.195:46133
I20260929 10:58:33.192855 30248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.195:46133 every 8 connection(s)
I20260929 10:58:33.193193 30121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
I20260929 10:58:33.197342 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 30121
I20260929 10:58:33.197455 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/ts-2/wal/instance
I20260929 10:58:33.197672 11195 external_mini_cluster.cc:1428] Setting key d7ccd50e06ea421d482b8a991a223ce0
I20260929 10:58:33.198652 29853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:33.193923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:39085 (local address 127.10.238.254:37233)
0929 10:58:33.193974 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:33.193977 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:33.194263 (+ 286us) server_negotiation.cc:407] Connection header received
0929 10:58:33.194763 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:33.194765 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:33.194779 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:33.194833 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:33.195553 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:33.196054 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:33.197029 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:33.197232 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:33.197357 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:33.198345 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:33.198347 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:33.198347 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:58:33.198352 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:33.198369 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:58:33.198374 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:33.198433 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:33.198530 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 10:58:33.198562 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 10:58:33.198885 30251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:33.193935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:37233 (local address 127.10.238.195:39085)
0929 10:58:33.194137 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:33.194171 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 10:58:33.194669 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:58:33.194915 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:58:33.194918 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:58:33.194926 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:58:33.195475 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:33.195480 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:33.196134 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:33.196135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:33.196959 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:33.196961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:33.197369 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:33.197370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:33.197561 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:33.198175 (+ 614us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:58:33.198176 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:58:33.198231 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:58:33.198259 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:58:33.198442 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:58:33.198449 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:58:33.198452 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 10:58:33.198523 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 10:58:33.198634 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":91,"threads_started":1}
I20260929 10:58:33.199319 30249 heartbeater.cc:347] Connected to a master server at 127.10.238.254:37233
I20260929 10:58:33.199395 30249 heartbeater.cc:464] Registering TS with master...
I20260929 10:58:33.199587 30249 heartbeater.cc:511] Master 127.10.238.254:37233 requested a full tablet report, sending...
I20260929 10:58:33.199961 29799 ts_manager.cc:194] Registered new tserver with Master: 2cc64fe379fd497b8063ab4786988009 (127.10.238.195:46133)
I20260929 10:58:33.200323 29799 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.238.195:39085
I20260929 10:58:33.209503 11195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:58:33.214223 29853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:33.210918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48362 (local address 127.10.238.254:37233)
0929 10:58:33.210964 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:33.210970 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:33.211090 (+ 120us) server_negotiation.cc:407] Connection header received
0929 10:58:33.211195 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:33.211196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:33.211206 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:33.211279 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:33.211487 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:33.211939 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:33.212929 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:33.213106 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:33.213160 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:33.213926 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:33.213927 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:33.213927 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:58:33.213931 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:33.213944 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:58:33.213947 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:33.214017 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:33.214150 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 10:58:33.214161 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 10:58:33.215703 29799 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48362:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:58:33.217243 29799 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:58:33.224066 30262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:33.219827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46030 (local address 127.10.238.195:46133)
0929 10:58:33.219988 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:33.219993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:33.220002 (+ 9us) server_negotiation.cc:407] Connection header received
0929 10:58:33.220041 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:33.220050 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:33.220062 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:33.220152 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:58:33.220314 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:33.221523 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:33.223144 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:33.223635 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:33.223732 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:33.223838 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:33.223922 (+ 84us) server_negotiation.cc:299] Negotiation successful
0929 10:58:33.223967 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":75,"threads_started":1}
I20260929 10:58:33.224342 30258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:33.219481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.194:34429 (local address 127.0.0.1:58232)
0929 10:58:33.219649 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:33.219716 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0929 10:58:33.219789 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:58:33.220277 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:58:33.220278 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:58:33.220281 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:58:33.220327 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:33.220328 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:33.221621 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:33.221621 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:33.223041 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:33.223044 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:33.224102 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:33.224103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:33.224166 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:33.224170 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 10:58:33.224220 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 10:58:33.224263 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":72,"thread_start_us":62,"threads_started":1}
I20260929 10:58:33.224673 30260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:33.219648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58232 (local address 127.10.238.194:34429)
0929 10:58:33.220014 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:33.220018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:33.220027 (+ 9us) server_negotiation.cc:407] Connection header received
0929 10:58:33.220064 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:33.220066 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:33.220101 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:33.220187 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:58:33.220413 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:33.221521 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:33.223132 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:33.223866 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:33.223976 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:33.224066 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:33.224542 (+ 476us) server_negotiation.cc:299] Negotiation successful
0929 10:58:33.224590 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":86,"threads_started":1}
I20260929 10:58:33.225028 30259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:33.219481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:35623 (local address 127.0.0.1:58244)
0929 10:58:33.219740 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:33.219749 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 10:58:33.219789 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:58:33.221544 (+ 1755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:58:33.221545 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:58:33.221548 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:58:33.221587 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:33.221588 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:33.222681 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:33.222683 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:33.223629 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:33.223632 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:33.224809 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:33.224810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:33.224872 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:33.224876 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 10:58:33.224906 (+ 30us) client_negotiation.cc:240] Negotiation successful
0929 10:58:33.224954 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":3,"spinlock_wait_cycles":768,"thread_start_us":55,"threads_started":1}
I20260929 10:58:33.225505 30261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:33.219756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58244 (local address 127.10.238.193:35623)
0929 10:58:33.221394 (+ 1638us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:33.221398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:33.221406 (+ 8us) server_negotiation.cc:407] Connection header received
0929 10:58:33.221438 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:33.221439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:33.221451 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:33.221544 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:58:33.221646 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:33.222599 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:33.223734 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:33.224708 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:33.225241 (+ 533us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:33.225388 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:33.225411 (+ 23us) server_negotiation.cc:299] Negotiation successful
0929 10:58:33.225443 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1600,"thread_start_us":87,"threads_started":1}
I20260929 10:58:33.225445 30052 tablet_service.cc:1467] Processing CreateTablet for tablet a44d090c8f3e4d17bc237ea4d2879d85 (DEFAULT_TABLE table=test-table [id=f2f0148ce1864d71b5c03da64c124eb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:58:33.225795 30052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a44d090c8f3e4d17bc237ea4d2879d85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:58:33.226140 29921 tablet_service.cc:1467] Processing CreateTablet for tablet a44d090c8f3e4d17bc237ea4d2879d85 (DEFAULT_TABLE table=test-table [id=f2f0148ce1864d71b5c03da64c124eb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:58:33.226454 29921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a44d090c8f3e4d17bc237ea4d2879d85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:58:33.229019 30263 tablet_bootstrap.cc:492] T a44d090c8f3e4d17bc237ea4d2879d85 P ad2af08371944acba50413b03ebf9ed4: Bootstrap starting.
I20260929 10:58:33.229285 30264 tablet_bootstrap.cc:492] T a44d090c8f3e4d17bc237ea4d2879d85 P 216d2fe7069f4712b0a28d358f3d187b: Bootstrap starting.
I20260929 10:58:33.229779 30263 tablet_bootstrap.cc:654] T a44d090c8f3e4d17bc237ea4d2879d85 P ad2af08371944acba50413b03ebf9ed4: Neither blocks nor log segments found. Creating new log.
I20260929 10:58:33.229874 30264 tablet_bootstrap.cc:654] T a44d090c8f3e4d17bc237ea4d2879d85 P 216d2fe7069f4712b0a28d358f3d187b: Neither blocks nor log segments found. Creating new log.
I20260929 10:58:33.230101 30263 log.cc:824] T a44d090c8f3e4d17bc237ea4d2879d85 P ad2af08371944acba50413b03ebf9ed4: Log is configured to *not* fsync() on all Append() calls
I20260929 10:58:33.230150 30264 log.cc:824] T a44d090c8f3e4d17bc237ea4d2879d85 P 216d2fe7069f4712b0a28d358f3d187b: Log is configured to *not* fsync() on all Append() calls
I20260929 10:58:33.230665 30257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:33.219490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.195:46133 (local address 127.0.0.1:46030)
0929 10:58:33.219888 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:33.219896 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 10:58:33.219932 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:58:33.220154 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:58:33.220157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:58:33.220159 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:58:33.220236 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:33.220246 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:33.221620 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:33.221621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:33.223041 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:33.223044 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:33.223728 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:33.223729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:33.223819 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:33.223833 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 10:58:33.223909 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 10:58:33.230588 (+ 6679us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"mutex_wait_us":70,"thread_start_us":56,"threads_started":1}
I20260929 10:58:33.231304 30264 tablet_bootstrap.cc:492] T a44d090c8f3e4d17bc237ea4d2879d85 P 216d2fe7069f4712b0a28d358f3d187b: No bootstrap required, opened a new log
I20260929 10:58:33.231359 30264 ts_tablet_manager.cc:1397] T a44d090c8f3e4d17bc237ea4d2879d85 P 216d2fe7069f4712b0a28d358f3d187b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 10:58:33.231333 30263 tablet_bootstrap.cc:492] T a44d090c8f3e4d17bc237ea4d2879d85 P ad2af08371944acba50413b03ebf9ed4: No bootstrap required, opened a new log
I20260929 10:58:33.231527 30263 ts_tablet_manager.cc:1397] T a44d090c8f3e4d17bc237ea4d2879d85 P ad2af08371944acba50413b03ebf9ed4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 10:58:33.231449 30183 tablet_service.cc:1467] Processing CreateTablet for tablet a44d090c8f3e4d17bc237ea4d2879d85 (DEFAULT_TABLE table=test-table [id=f2f0148ce1864d71b5c03da64c124eb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:58:33.231794 30183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a44d090c8f3e4d17bc237ea4d2879d85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:58:33.233206 30264 raft_consensus.cc:348] T a44d090c8f3e4d17bc237ea4d2879d85 P 216d2fe7069f4712b0a28d358f3d187b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2af08371944acba50413b03ebf9ed4" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 34429 } } peers { permanent_uuid: "216d2fe7069f4712b0a28d358f3d187b" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 35623 } } peers { permanent_uuid: "2cc64fe379fd497b8063ab4786988009" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 46133 } }
I20260929 10:58:33.233431 30264 raft_consensus.cc:374] T a44d090c8f3e4d17bc237ea4d2879d85 P 216d2fe7069f4712b0a28d358f3d187b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:58:33.233474 30264 raft_consensus.cc:729] T a44d090c8f3e4d17bc237ea4d2879d85 P 216d2fe7069f4712b0a28d358f3d187b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 216d2fe7069f4712b0a28d358f3d187b, State: Initialized, Role: FOLLOWER
I20260929 10:58:33.233615 30264 consensus_queue.cc:260] T a44d090c8f3e4d17bc237ea4d2879d85 P 216d2fe7069f4712b0a28d358f3d187b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2af08371944acba50413b03ebf9ed4" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 34429 } } peers { permanent_uuid: "216d2fe7069f4712b0a28d358f3d187b" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 35623 } } peers { permanent_uuid: "2cc64fe379fd497b8063ab4786988009" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 46133 } }
I20260929 10:58:33.233911 30264 ts_tablet_manager.cc:1428] T a44d090c8f3e4d17bc237ea4d2879d85 P 216d2fe7069f4712b0a28d358f3d187b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 10:58:33.233924 29987 heartbeater.cc:503] Master 127.10.238.254:37233 was elected leader, sending a full tablet report...
I20260929 10:58:33.234177 30263 raft_consensus.cc:348] T a44d090c8f3e4d17bc237ea4d2879d85 P ad2af08371944acba50413b03ebf9ed4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2af08371944acba50413b03ebf9ed4" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 34429 } } peers { permanent_uuid: "216d2fe7069f4712b0a28d358f3d187b" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 35623 } } peers { permanent_uuid: "2cc64fe379fd497b8063ab4786988009" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 46133 } }
I20260929 10:58:33.234342 30263 raft_consensus.cc:374] T a44d090c8f3e4d17bc237ea4d2879d85 P ad2af08371944acba50413b03ebf9ed4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:58:33.234385 30263 raft_consensus.cc:729] T a44d090c8f3e4d17bc237ea4d2879d85 P ad2af08371944acba50413b03ebf9ed4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2af08371944acba50413b03ebf9ed4, State: Initialized, Role: FOLLOWER
I20260929 10:58:33.234488 30263 consensus_queue.cc:260] T a44d090c8f3e4d17bc237ea4d2879d85 P ad2af08371944acba50413b03ebf9ed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2af08371944acba50413b03ebf9ed4" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 34429 } } peers { permanent_uuid: "216d2fe7069f4712b0a28d358f3d187b" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 35623 } } peers { permanent_uuid: "2cc64fe379fd497b8063ab4786988009" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 46133 } }
I20260929 10:58:33.234741 30263 ts_tablet_manager.cc:1428] T a44d090c8f3e4d17bc237ea4d2879d85 P ad2af08371944acba50413b03ebf9ed4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 10:58:33.234772 30118 heartbeater.cc:503] Master 127.10.238.254:37233 was elected leader, sending a full tablet report...
I20260929 10:58:33.235517 30269 tablet_bootstrap.cc:492] T a44d090c8f3e4d17bc237ea4d2879d85 P 2cc64fe379fd497b8063ab4786988009: Bootstrap starting.
I20260929 10:58:33.236099 30269 tablet_bootstrap.cc:654] T a44d090c8f3e4d17bc237ea4d2879d85 P 2cc64fe379fd497b8063ab4786988009: Neither blocks nor log segments found. Creating new log.
I20260929 10:58:33.236424 30269 log.cc:824] T a44d090c8f3e4d17bc237ea4d2879d85 P 2cc64fe379fd497b8063ab4786988009: Log is configured to *not* fsync() on all Append() calls
I20260929 10:58:33.237067 30269 tablet_bootstrap.cc:492] T a44d090c8f3e4d17bc237ea4d2879d85 P 2cc64fe379fd497b8063ab4786988009: No bootstrap required, opened a new log
I20260929 10:58:33.237149 30269 ts_tablet_manager.cc:1397] T a44d090c8f3e4d17bc237ea4d2879d85 P 2cc64fe379fd497b8063ab4786988009: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 10:58:33.238924 30269 raft_consensus.cc:348] T a44d090c8f3e4d17bc237ea4d2879d85 P 2cc64fe379fd497b8063ab4786988009 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2af08371944acba50413b03ebf9ed4" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 34429 } } peers { permanent_uuid: "216d2fe7069f4712b0a28d358f3d187b" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 35623 } } peers { permanent_uuid: "2cc64fe379fd497b8063ab4786988009" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 46133 } }
I20260929 10:58:33.239071 30269 raft_consensus.cc:374] T a44d090c8f3e4d17bc237ea4d2879d85 P 2cc64fe379fd497b8063ab4786988009 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:58:33.239117 30269 raft_consensus.cc:729] T a44d090c8f3e4d17bc237ea4d2879d85 P 2cc64fe379fd497b8063ab4786988009 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cc64fe379fd497b8063ab4786988009, State: Initialized, Role: FOLLOWER
I20260929 10:58:33.239238 30269 consensus_queue.cc:260] T a44d090c8f3e4d17bc237ea4d2879d85 P 2cc64fe379fd497b8063ab4786988009 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2af08371944acba50413b03ebf9ed4" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 34429 } } peers { permanent_uuid: "216d2fe7069f4712b0a28d358f3d187b" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 35623 } } peers { permanent_uuid: "2cc64fe379fd497b8063ab4786988009" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 46133 } }
I20260929 10:58:33.239506 30269 ts_tablet_manager.cc:1428] T a44d090c8f3e4d17bc237ea4d2879d85 P 2cc64fe379fd497b8063ab4786988009: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260929 10:58:33.239543 30249 heartbeater.cc:503] Master 127.10.238.254:37233 was elected leader, sending a full tablet report...
I20260929 10:58:33.290803 30271 raft_consensus.cc:482] T a44d090c8f3e4d17bc237ea4d2879d85 P 2cc64fe379fd497b8063ab4786988009 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:58:33.290941 30271 raft_consensus.cc:504] T a44d090c8f3e4d17bc237ea4d2879d85 P 2cc64fe379fd497b8063ab4786988009 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2af08371944acba50413b03ebf9ed4" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 34429 } } peers { permanent_uuid: "216d2fe7069f4712b0a28d358f3d187b" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 35623 } } peers { permanent_uuid: "2cc64fe379fd497b8063ab4786988009" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 46133 } }
I20260929 10:58:33.291390 30271 leader_election.cc:288] T a44d090c8f3e4d17bc237ea4d2879d85 P 2cc64fe379fd497b8063ab4786988009 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad2af08371944acba50413b03ebf9ed4 (127.10.238.194:34429), 216d2fe7069f4712b0a28d358f3d187b (127.10.238.193:35623)
I20260929 10:58:33.294762 30261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:33.291722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:37303 (local address 127.10.238.193:35623)
0929 10:58:33.291784 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:33.291785 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:33.291797 (+ 12us) server_negotiation.cc:407] Connection header received
0929 10:58:33.291871 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:33.291872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:33.291974 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:33.292079 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:58:33.292248 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:33.292927 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:33.294014 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:33.294382 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:33.294467 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:33.294476 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:33.294626 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 10:58:33.294666 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 10:58:33.294786 30251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:33.291611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:35623 (local address 127.10.238.195:37303)
0929 10:58:33.291684 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:33.291699 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:58:33.291805 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:58:33.292086 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:58:33.292087 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:58:33.292089 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:58:33.292170 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:33.292172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:33.293010 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:33.293011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:33.293912 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:33.293914 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:33.294465 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:33.294466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:33.294543 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:33.294545 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 10:58:33.294619 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 10:58:33.294665 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":29}
I20260929 10:58:33.294776 30260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:33.291738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:40301 (local address 127.10.238.194:34429)
0929 10:58:33.291787 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:33.291789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:33.291951 (+ 162us) server_negotiation.cc:407] Connection header received
0929 10:58:33.292027 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:33.292029 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:33.292042 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:33.292088 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:58:33.292269 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:33.292933 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:33.294009 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:33.294383 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:33.294480 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:33.294489 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:33.294658 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 10:58:33.294700 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 10:58:33.294872 30272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:33.291618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.194:34429 (local address 127.10.238.195:40301)
0929 10:58:33.291868 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:33.291878 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 10:58:33.291970 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:58:33.292099 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:58:33.292099 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:58:33.292101 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:58:33.292184 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:33.292186 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:33.293015 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:33.293016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:33.293912 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:33.293915 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:33.294489 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:33.294491 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:33.294564 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:33.294568 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 10:58:33.294649 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 10:58:33.294690 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":69,"thread_start_us":116,"threads_started":1}
I20260929 10:58:33.295225 29941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a44d090c8f3e4d17bc237ea4d2879d85" candidate_uuid: "2cc64fe379fd497b8063ab4786988009" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "216d2fe7069f4712b0a28d358f3d187b" is_pre_election: true
I20260929 10:58:33.295264 30072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a44d090c8f3e4d17bc237ea4d2879d85" candidate_uuid: "2cc64fe379fd497b8063ab4786988009" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2af08371944acba50413b03ebf9ed4" is_pre_election: true
I20260929 10:58:33.295418 29941 raft_consensus.cc:2454] T a44d090c8f3e4d17bc237ea4d2879d85 P 216d2fe7069f4712b0a28d358f3d187b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cc64fe379fd497b8063ab4786988009 in term 0.
I20260929 10:58:33.295431 30072 raft_consensus.cc:2454] T a44d090c8f3e4d17bc237ea4d2879d85 P ad2af08371944acba50413b03ebf9ed4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cc64fe379fd497b8063ab4786988009 in term 0.
I20260929 10:58:33.295693 30137 leader_election.cc:302] T a44d090c8f3e4d17bc237ea4d2879d85 P 2cc64fe379fd497b8063ab4786988009 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cc64fe379fd497b8063ab4786988009, ad2af08371944acba50413b03ebf9ed4; no voters:
I20260929 10:58:33.295851 30271 raft_consensus.cc:2781] T a44d090c8f3e4d17bc237ea4d2879d85 P 2cc64fe379fd497b8063ab4786988009 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:58:33.295910 30271 raft_consensus.cc:482] T a44d090c8f3e4d17bc237ea4d2879d85 P 2cc64fe379fd497b8063ab4786988009 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:58:33.295931 30271 raft_consensus.cc:3037] T a44d090c8f3e4d17bc237ea4d2879d85 P 2cc64fe379fd497b8063ab4786988009 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:58:33.296712 30271 raft_consensus.cc:504] T a44d090c8f3e4d17bc237ea4d2879d85 P 2cc64fe379fd497b8063ab4786988009 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2af08371944acba50413b03ebf9ed4" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 34429 } } peers { permanent_uuid: "216d2fe7069f4712b0a28d358f3d187b" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 35623 } } peers { permanent_uuid: "2cc64fe379fd497b8063ab4786988009" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 46133 } }
I20260929 10:58:33.296887 30271 leader_election.cc:288] T a44d090c8f3e4d17bc237ea4d2879d85 P 2cc64fe379fd497b8063ab4786988009 [CANDIDATE]: Term 1 election: Requested vote from peers ad2af08371944acba50413b03ebf9ed4 (127.10.238.194:34429), 216d2fe7069f4712b0a28d358f3d187b (127.10.238.193:35623)
I20260929 10:58:33.297040 29941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a44d090c8f3e4d17bc237ea4d2879d85" candidate_uuid: "2cc64fe379fd497b8063ab4786988009" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "216d2fe7069f4712b0a28d358f3d187b"
I20260929 10:58:33.297034 30072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a44d090c8f3e4d17bc237ea4d2879d85" candidate_uuid: "2cc64fe379fd497b8063ab4786988009" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2af08371944acba50413b03ebf9ed4"
W20260929 10:58:33.297048 30119 tablet.cc:2369] T a44d090c8f3e4d17bc237ea4d2879d85 P ad2af08371944acba50413b03ebf9ed4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:58:33.297122 29941 raft_consensus.cc:3037] T a44d090c8f3e4d17bc237ea4d2879d85 P 216d2fe7069f4712b0a28d358f3d187b [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:58:33.297148 30072 raft_consensus.cc:3037] T a44d090c8f3e4d17bc237ea4d2879d85 P ad2af08371944acba50413b03ebf9ed4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:58:33.297938 30072 raft_consensus.cc:2454] T a44d090c8f3e4d17bc237ea4d2879d85 P ad2af08371944acba50413b03ebf9ed4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cc64fe379fd497b8063ab4786988009 in term 1.
I20260929 10:58:33.297938 29941 raft_consensus.cc:2454] T a44d090c8f3e4d17bc237ea4d2879d85 P 216d2fe7069f4712b0a28d358f3d187b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cc64fe379fd497b8063ab4786988009 in term 1.
I20260929 10:58:33.298130 30139 leader_election.cc:302] T a44d090c8f3e4d17bc237ea4d2879d85 P 2cc64fe379fd497b8063ab4786988009 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 216d2fe7069f4712b0a28d358f3d187b, 2cc64fe379fd497b8063ab4786988009; no voters:
I20260929 10:58:33.298240 30271 raft_consensus.cc:2781] T a44d090c8f3e4d17bc237ea4d2879d85 P 2cc64fe379fd497b8063ab4786988009 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:58:33.298398 30271 raft_consensus.cc:686] T a44d090c8f3e4d17bc237ea4d2879d85 P 2cc64fe379fd497b8063ab4786988009 [term 1 LEADER]: Becoming Leader. State: Replica: 2cc64fe379fd497b8063ab4786988009, State: Running, Role: LEADER
I20260929 10:58:33.298504 30271 consensus_queue.cc:237] T a44d090c8f3e4d17bc237ea4d2879d85 P 2cc64fe379fd497b8063ab4786988009 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2af08371944acba50413b03ebf9ed4" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 34429 } } peers { permanent_uuid: "216d2fe7069f4712b0a28d358f3d187b" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 35623 } } peers { permanent_uuid: "2cc64fe379fd497b8063ab4786988009" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 46133 } }
I20260929 10:58:33.299351 29799 catalog_manager.cc:5495] T a44d090c8f3e4d17bc237ea4d2879d85 P 2cc64fe379fd497b8063ab4786988009 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2cc64fe379fd497b8063ab4786988009 (127.10.238.195). New cstate: current_term: 1 leader_uuid: "2cc64fe379fd497b8063ab4786988009" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2af08371944acba50413b03ebf9ed4" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 34429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "216d2fe7069f4712b0a28d358f3d187b" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 35623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cc64fe379fd497b8063ab4786988009" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 46133 } health_report { overall_health: HEALTHY } } }
I20260929 10:58:33.316119 30262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:33.313850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46044 (local address 127.10.238.195:46133)
0929 10:58:33.313894 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:33.313896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:33.313906 (+ 10us) server_negotiation.cc:407] Connection header received
0929 10:58:33.313952 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:33.313952 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:33.313973 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:33.314043 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:58:33.314217 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:33.314702 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:33.315452 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:33.315631 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:33.315701 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:33.315830 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:58:33.315914 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:58:33.315983 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:33.316026 (+ 43us) server_negotiation.cc:299] Negotiation successful
0929 10:58:33.316034 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 10:58:33.318204 29941 raft_consensus.cc:1261] T a44d090c8f3e4d17bc237ea4d2879d85 P 216d2fe7069f4712b0a28d358f3d187b [term 1 FOLLOWER]: Refusing update from remote peer 2cc64fe379fd497b8063ab4786988009: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:58:33.318234 30072 raft_consensus.cc:1261] T a44d090c8f3e4d17bc237ea4d2879d85 P ad2af08371944acba50413b03ebf9ed4 [term 1 FOLLOWER]: Refusing update from remote peer 2cc64fe379fd497b8063ab4786988009: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:58:33.318608 30273 consensus_queue.cc:1035] T a44d090c8f3e4d17bc237ea4d2879d85 P 2cc64fe379fd497b8063ab4786988009 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad2af08371944acba50413b03ebf9ed4" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 34429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:58:33.318763 30271 consensus_queue.cc:1035] T a44d090c8f3e4d17bc237ea4d2879d85 P 2cc64fe379fd497b8063ab4786988009 [LEADER]: Connected to new peer: Peer: permanent_uuid: "216d2fe7069f4712b0a28d358f3d187b" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 35623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:58:33.321205 30276 mvcc.cc:205] Tried to move back new op lower bound from 7334623286548258816 to 7334623286471372800. Current Snapshot: MvccSnapshot[applied={T|T < 7334623286548258816}]
I20260929 10:58:33.321805 30277 mvcc.cc:205] Tried to move back new op lower bound from 7334623286548258816 to 7334623286471372800. Current Snapshot: MvccSnapshot[applied={T|T < 7334623286548258816}]
I20260929 10:58:33.323958 29798 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:58:33.324102 29798 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:58:33.324926 29798 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 526ccc0ae8f145659bee9181f2cd9825: Sending DeleteTablet for 3 replicas of tablet a44d090c8f3e4d17bc237ea4d2879d85
I20260929 10:58:33.325294 30183 tablet_service.cc:1514] Processing DeleteTablet for tablet a44d090c8f3e4d17bc237ea4d2879d85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:58:33 UTC) from {username='slave'} at 127.0.0.1:46030
I20260929 10:58:33.325331 30052 tablet_service.cc:1514] Processing DeleteTablet for tablet a44d090c8f3e4d17bc237ea4d2879d85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:58:33 UTC) from {username='slave'} at 127.0.0.1:58232
I20260929 10:58:33.325369 29921 tablet_service.cc:1514] Processing DeleteTablet for tablet a44d090c8f3e4d17bc237ea4d2879d85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:58:33 UTC) from {username='slave'} at 127.0.0.1:58244
I20260929 10:58:33.325668 30287 tablet_replica.cc:330] T a44d090c8f3e4d17bc237ea4d2879d85 P ad2af08371944acba50413b03ebf9ed4: stopping tablet replica
I20260929 10:58:33.325688 30286 tablet_replica.cc:330] T a44d090c8f3e4d17bc237ea4d2879d85 P 2cc64fe379fd497b8063ab4786988009: stopping tablet replica
I20260929 10:58:33.325783 30287 raft_consensus.cc:2229] T a44d090c8f3e4d17bc237ea4d2879d85 P ad2af08371944acba50413b03ebf9ed4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:58:33.325839 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 29859
I20260929 10:58:33.325891 30287 raft_consensus.cc:2258] T a44d090c8f3e4d17bc237ea4d2879d85 P ad2af08371944acba50413b03ebf9ed4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:58:33.326718 30286 raft_consensus.cc:2229] T a44d090c8f3e4d17bc237ea4d2879d85 P 2cc64fe379fd497b8063ab4786988009 [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:58:33.326879 30286 raft_consensus.cc:2258] T a44d090c8f3e4d17bc237ea4d2879d85 P 2cc64fe379fd497b8063ab4786988009 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:58:33.327061 30287 ts_tablet_manager.cc:1905] T a44d090c8f3e4d17bc237ea4d2879d85 P ad2af08371944acba50413b03ebf9ed4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:58:33.327428 30286 ts_tablet_manager.cc:1905] T a44d090c8f3e4d17bc237ea4d2879d85 P 2cc64fe379fd497b8063ab4786988009: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:58:33.328902 30287 ts_tablet_manager.cc:1918] T a44d090c8f3e4d17bc237ea4d2879d85 P ad2af08371944acba50413b03ebf9ed4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:58:33.328980 30287 log.cc:1196] T a44d090c8f3e4d17bc237ea4d2879d85 P ad2af08371944acba50413b03ebf9ed4: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/ts-1/wal/wals/a44d090c8f3e4d17bc237ea4d2879d85
I20260929 10:58:33.329242 30287 ts_tablet_manager.cc:1939] T a44d090c8f3e4d17bc237ea4d2879d85 P ad2af08371944acba50413b03ebf9ed4: Deleting consensus metadata
I20260929 10:58:33.329512 30286 ts_tablet_manager.cc:1918] T a44d090c8f3e4d17bc237ea4d2879d85 P 2cc64fe379fd497b8063ab4786988009: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:58:33.329633 30286 log.cc:1196] T a44d090c8f3e4d17bc237ea4d2879d85 P 2cc64fe379fd497b8063ab4786988009: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790679274510869-11195-0/minicluster-data/ts-2/wal/wals/a44d090c8f3e4d17bc237ea4d2879d85
I20260929 10:58:33.329676 29784 catalog_manager.cc:4841] TS ad2af08371944acba50413b03ebf9ed4 (127.10.238.194:34429): tablet a44d090c8f3e4d17bc237ea4d2879d85 (table test-table [id=f2f0148ce1864d71b5c03da64c124eb7]) successfully deleted
I20260929 10:58:33.329953 30286 ts_tablet_manager.cc:1939] T a44d090c8f3e4d17bc237ea4d2879d85 P 2cc64fe379fd497b8063ab4786988009: Deleting consensus metadata
I20260929 10:58:33.330327 29783 catalog_manager.cc:4841] TS 2cc64fe379fd497b8063ab4786988009 (127.10.238.195:46133): tablet a44d090c8f3e4d17bc237ea4d2879d85 (table test-table [id=f2f0148ce1864d71b5c03da64c124eb7]) successfully deleted
W20260929 10:58:33.334092 29786 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.238.193:35623 (error 108)
I20260929 10:58:33.334311 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 29990
I20260929 10:58:33.334604 30257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:33.334486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:35623 (local address 127.0.0.1:58250)
0929 10:58:33.334532 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:33.334566 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.193:35623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260929 10:58:33.334697 29786 catalog_manager.cc:4584] TS 216d2fe7069f4712b0a28d358f3d187b (127.10.238.193:35623): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a44d090c8f3e4d17bc237ea4d2879d85: Network error: Client connection negotiation failed: client connection to 127.10.238.193:35623: connect: Connection refused (error 111)
I20260929 10:58:33.339372 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 30121
I20260929 10:58:33.344292 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 29768
2026-09-29T10:58:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (784 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:58:33 dist-test-slave-8nrt krb5kdc[30292](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:58:33 dist-test-slave-8nrt krb5kdc[30292](info): set up 2 sockets
Sep 29 10:58:33 dist-test-slave-8nrt krb5kdc[30292](info): commencing operation
krb5kdc: starting...
W20260929 10:58:34.381179 11195 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Sep 29 10:58:34 dist-test-slave-8nrt krb5kdc[30292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:58:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:58:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
I20260929 10:58:34.499825 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:41227
--webserver_interface=127.10.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:40269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:41227
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:58:34.580824 30305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:58:34.581025 30305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:58:34.581048 30305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:58:34.582260 30305 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:58:34.582304 30305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:58:34.582319 30305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:58:34.582331 30305 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:58:34.582343 30305 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:58:34.583848 30305 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:40269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:41227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:41227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:58:34.584111 30305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:58:34.584384 30305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:58:34.589495 30313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:34.589546 30305 server_base.cc:1034] running on GCE node
W20260929 10:58:34.589603 30310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:34.589496 30311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:34.589920 30305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:58:34.590161 30305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:58:34.591331 30305 hybrid_clock.cc:648] HybridClock initialized: now 1790679514591309 us; error 34 us; skew 500 ppm
Sep 29 10:58:34 dist-test-slave-8nrt krb5kdc[30292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679514, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:58:34.600013 30305 init.cc:377] Logged in from keytab as kudu/127.10.238.254@KRBTEST.COM (short username kudu)
I20260929 10:58:34.600487 30305 webserver.cc:460] Webserver started at http://127.10.238.254:41731/ using document root <none> and password file <none>
I20260929 10:58:34.600680 30305 fs_manager.cc:359] Metadata directory not provided
I20260929 10:58:34.600720 30305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:58:34.600817 30305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:58:34.601879 30305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/master-0/data/instance:
uuid: "2ce90d4b5e6d400181ab6d655379a2a4"
format_stamp: "Formatted at 2026-09-29 10:58:34 on dist-test-slave-8nrt"
server_key: "5cbff8cabbf2325cc646339c6f7f68e4"
server_key_iv: "1410f47e00a2f4a68a2865ce7762c284"
server_key_version: "encryptionkey@0"
I20260929 10:58:34.602181 30305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/master-0/wal/instance:
uuid: "2ce90d4b5e6d400181ab6d655379a2a4"
format_stamp: "Formatted at 2026-09-29 10:58:34 on dist-test-slave-8nrt"
server_key: "5cbff8cabbf2325cc646339c6f7f68e4"
server_key_iv: "1410f47e00a2f4a68a2865ce7762c284"
server_key_version: "encryptionkey@0"
I20260929 10:58:34.603439 30305 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 10:58:34.604136 30320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:34.604332 30305 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:58:34.604415 30305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "2ce90d4b5e6d400181ab6d655379a2a4"
format_stamp: "Formatted at 2026-09-29 10:58:34 on dist-test-slave-8nrt"
server_key: "5cbff8cabbf2325cc646339c6f7f68e4"
server_key_iv: "1410f47e00a2f4a68a2865ce7762c284"
server_key_version: "encryptionkey@0"
I20260929 10:58:34.604468 30305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:58:34.611161 30305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:58:34.613396 30305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:58:34.613555 30305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:58:34.617426 30305 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.254:41227
I20260929 10:58:34.617497 30372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.254:41227 every 8 connection(s)
I20260929 10:58:34.617820 30305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:58:34.618659 30373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:58:34.620792 30373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ce90d4b5e6d400181ab6d655379a2a4: Bootstrap starting.
I20260929 10:58:34.621367 30373 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ce90d4b5e6d400181ab6d655379a2a4: Neither blocks nor log segments found. Creating new log.
I20260929 10:58:34.621609 30373 log.cc:824] T 00000000000000000000000000000000 P 2ce90d4b5e6d400181ab6d655379a2a4: Log is configured to *not* fsync() on all Append() calls
I20260929 10:58:34.622252 30373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ce90d4b5e6d400181ab6d655379a2a4: No bootstrap required, opened a new log
I20260929 10:58:34.623768 30373 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ce90d4b5e6d400181ab6d655379a2a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ce90d4b5e6d400181ab6d655379a2a4" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 41227 } }
I20260929 10:58:34.623900 30373 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ce90d4b5e6d400181ab6d655379a2a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:58:34.623922 30373 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ce90d4b5e6d400181ab6d655379a2a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ce90d4b5e6d400181ab6d655379a2a4, State: Initialized, Role: FOLLOWER
I20260929 10:58:34.624020 30373 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ce90d4b5e6d400181ab6d655379a2a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ce90d4b5e6d400181ab6d655379a2a4" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 41227 } }
I20260929 10:58:34.624102 30373 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ce90d4b5e6d400181ab6d655379a2a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:58:34.624140 30373 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ce90d4b5e6d400181ab6d655379a2a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:58:34.624192 30373 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ce90d4b5e6d400181ab6d655379a2a4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:58:34.624368 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 30305
I20260929 10:58:34.624487 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/master-0/wal/instance
I20260929 10:58:34.624729 11195 external_mini_cluster.cc:1428] Setting key 7695d2e091d81876ec6c19b6455542ce
I20260929 10:58:34.624908 30373 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ce90d4b5e6d400181ab6d655379a2a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ce90d4b5e6d400181ab6d655379a2a4" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 41227 } }
I20260929 10:58:34.625029 30373 leader_election.cc:302] T 00000000000000000000000000000000 P 2ce90d4b5e6d400181ab6d655379a2a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ce90d4b5e6d400181ab6d655379a2a4; no voters:
I20260929 10:58:34.625202 30373 leader_election.cc:288] T 00000000000000000000000000000000 P 2ce90d4b5e6d400181ab6d655379a2a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:58:34.625276 30376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ce90d4b5e6d400181ab6d655379a2a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:58:34.625427 30376 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ce90d4b5e6d400181ab6d655379a2a4 [term 1 LEADER]: Becoming Leader. State: Replica: 2ce90d4b5e6d400181ab6d655379a2a4, State: Running, Role: LEADER
I20260929 10:58:34.625500 30373 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ce90d4b5e6d400181ab6d655379a2a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:58:34.625526 30376 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ce90d4b5e6d400181ab6d655379a2a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ce90d4b5e6d400181ab6d655379a2a4" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 41227 } }
I20260929 10:58:34.625859 30376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ce90d4b5e6d400181ab6d655379a2a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ce90d4b5e6d400181ab6d655379a2a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ce90d4b5e6d400181ab6d655379a2a4" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 41227 } } }
I20260929 10:58:34.625936 30376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ce90d4b5e6d400181ab6d655379a2a4 [sys.catalog]: This master's current role is: LEADER
I20260929 10:58:34.626070 30376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ce90d4b5e6d400181ab6d655379a2a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ce90d4b5e6d400181ab6d655379a2a4. Latest consensus state: current_term: 1 leader_uuid: "2ce90d4b5e6d400181ab6d655379a2a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ce90d4b5e6d400181ab6d655379a2a4" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 41227 } } }
I20260929 10:58:34.626119 30376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ce90d4b5e6d400181ab6d655379a2a4 [sys.catalog]: This master's current role is: LEADER
I20260929 10:58:34.626273 30385 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:58:34.626804 30385 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:58:34.628484 30385 catalog_manager.cc:1347] Generated new cluster ID: a710885f21f44692a6f60e6abbcd72ee
I20260929 10:58:34.628533 30385 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 10:58:34 dist-test-slave-8nrt krb5kdc[30292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:58:34.635643 30385 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:58:34.636399 30385 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:58:34.636847 30378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:34.625206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58742 (local address 127.10.238.254:41227)
0929 10:58:34.625521 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:34.625532 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:34.625569 (+ 37us) server_negotiation.cc:407] Connection header received
0929 10:58:34.625714 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:34.625722 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:34.625787 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:34.625970 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:34.626748 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:34.628161 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:34.629230 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:34.629844 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:34.629944 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:34.632667 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:34.632671 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:34.632671 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:58:34.632683 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:34.635282 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:34.635711 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:34.635714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:34.635715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:34.635820 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:34.636033 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:34.636034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:34.636035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:34.636270 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:34.636460 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:34.636529 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 10:58:34.636680 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":101,"threads_started":1}
I20260929 10:58:34.643812 30385 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ce90d4b5e6d400181ab6d655379a2a4: Generated new TSK 0
I20260929 10:58:34.643981 30385 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.238.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab.
Entry for principal kudu/127.10.238.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab.
I20260929 10:58:34.673815 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.193:0
--local_ip_for_outbound_sockets=127.10.238.193
--webserver_interface=127.10.238.193
--webserver_port=0
--tserver_master_addrs=127.10.238.254:41227
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab
--principal=kudu/127.10.238.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:40269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:58:34.757387 30399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:58:34.757627 30399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:58:34.757668 30399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:58:34.758862 30399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:58:34.758934 30399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:58:34.758980 30399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.193
I20260929 10:58:34.760488 30399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:40269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.193.keytab
--principal=kudu/127.10.238.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.238.193
--webserver_port=0
--tserver_master_addrs=127.10.238.254:41227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.193
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:58:34.760769 30399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:58:34.761004 30399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:58:34.766975 30405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:34.767035 30399 server_base.cc:1034] running on GCE node
W20260929 10:58:34.767100 30404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:34.766987 30407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:34.767493 30399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:58:34.767797 30399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:58:34.768970 30399 hybrid_clock.cc:648] HybridClock initialized: now 1790679514768956 us; error 42 us; skew 500 ppm
Sep 29 10:58:34 dist-test-slave-8nrt krb5kdc[30292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679514, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:58:34.771579 30399 init.cc:377] Logged in from keytab as kudu/127.10.238.193@KRBTEST.COM (short username kudu)
I20260929 10:58:34.772002 30399 webserver.cc:460] Webserver started at http://127.10.238.193:36879/ using document root <none> and password file <none>
I20260929 10:58:34.772249 30399 fs_manager.cc:359] Metadata directory not provided
I20260929 10:58:34.772320 30399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:58:34.772468 30399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:58:34.773613 30399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/ts-0/data/instance:
uuid: "5afb1c5268df4855b6941d8665d931a1"
format_stamp: "Formatted at 2026-09-29 10:58:34 on dist-test-slave-8nrt"
server_key: "aba630fa0a047c35b05b0a991827e0b1"
server_key_iv: "390b8618dd13613a0f104d6501db6267"
server_key_version: "encryptionkey@0"
I20260929 10:58:34.773972 30399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/ts-0/wal/instance:
uuid: "5afb1c5268df4855b6941d8665d931a1"
format_stamp: "Formatted at 2026-09-29 10:58:34 on dist-test-slave-8nrt"
server_key: "aba630fa0a047c35b05b0a991827e0b1"
server_key_iv: "390b8618dd13613a0f104d6501db6267"
server_key_version: "encryptionkey@0"
I20260929 10:58:34.775281 30399 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 10:58:34.775974 30414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:34.776149 30399 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:58:34.776250 30399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/ts-0/wal
uuid: "5afb1c5268df4855b6941d8665d931a1"
format_stamp: "Formatted at 2026-09-29 10:58:34 on dist-test-slave-8nrt"
server_key: "aba630fa0a047c35b05b0a991827e0b1"
server_key_iv: "390b8618dd13613a0f104d6501db6267"
server_key_version: "encryptionkey@0"
I20260929 10:58:34.776351 30399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:58:34.787889 30399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:58:34.790243 30399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:58:34.790405 30399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:58:34.790681 30399 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:58:34.791030 30399 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:58:34.791096 30399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:34.791127 30399 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:58:34.791157 30399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:34.798401 30399 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.193:33983
I20260929 10:58:34.798475 30527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.193:33983 every 8 connection(s)
I20260929 10:58:34.798834 30399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
Sep 29 10:58:34 dist-test-slave-8nrt krb5kdc[30292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679514, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.193@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:58:34.808646 30378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:34.799604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:47161 (local address 127.10.238.254:41227)
0929 10:58:34.799651 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:34.799653 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:34.799939 (+ 286us) server_negotiation.cc:407] Connection header received
0929 10:58:34.800508 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:34.800509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:34.800522 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:34.800579 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:34.801730 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:34.802270 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:34.803297 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:34.803515 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:34.803603 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:34.805814 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:34.805815 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:34.805816 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:58:34.805824 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:34.807154 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:34.807570 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:34.807572 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:34.807572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:34.807639 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:34.807952 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:34.807953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:34.807953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:34.808080 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:34.808164 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:34.808460 (+ 296us) server_negotiation.cc:299] Negotiation successful
0929 10:58:34.808554 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 10:58:34.808874 30530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:34.799595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:41227 (local address 127.10.238.193:47161)
0929 10:58:34.799795 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:34.799848 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 10:58:34.800399 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:58:34.800658 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:58:34.800662 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:58:34.801402 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:58:34.801607 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:34.801613 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:34.802379 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:34.802380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:34.803184 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:34.803187 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:34.803615 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:34.803616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:34.803749 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:34.804248 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:58:34.804257 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:58:34.805710 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:58:34.807250 (+ 1540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:58:34.807258 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:58:34.807260 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:58:34.807488 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:58:34.807709 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:58:34.807710 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:58:34.807711 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:58:34.807884 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:58:34.808164 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:58:34.808173 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:58:34.808318 (+ 145us) client_negotiation.cc:769] Sending connection context
0929 10:58:34.808428 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 10:58:34.808592 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":89,"threads_started":1}
I20260929 10:58:34.809068 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 30399
I20260929 10:58:34.809190 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/ts-0/wal/instance
I20260929 10:58:34.809448 11195 external_mini_cluster.cc:1428] Setting key 818c1ad0202e561f9a7120b3320dca9b
I20260929 10:58:34.809909 30528 heartbeater.cc:347] Connected to a master server at 127.10.238.254:41227
I20260929 10:58:34.810021 30528 heartbeater.cc:464] Registering TS with master...
I20260929 10:58:34.810225 30528 heartbeater.cc:511] Master 127.10.238.254:41227 requested a full tablet report, sending...
I20260929 10:58:34.810828 30337 ts_manager.cc:194] Registered new tserver with Master: 5afb1c5268df4855b6941d8665d931a1 (127.10.238.193:33983)
I20260929 10:58:34.811659 30337 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.193@KRBTEST.COM'} at 127.10.238.193:47161
WARNING: no policy specified for kudu/127.10.238.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab.
Entry for principal kudu/127.10.238.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab.
I20260929 10:58:34.835434 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.194:0
--local_ip_for_outbound_sockets=127.10.238.194
--webserver_interface=127.10.238.194
--webserver_port=0
--tserver_master_addrs=127.10.238.254:41227
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab
--principal=kudu/127.10.238.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:40269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:58:34.919112 30533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:58:34.919322 30533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:58:34.919348 30533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:58:34.920519 30533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:58:34.920562 30533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:58:34.920589 30533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.194
I20260929 10:58:34.922046 30533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:40269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.194.keytab
--principal=kudu/127.10.238.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.238.194
--webserver_port=0
--tserver_master_addrs=127.10.238.254:41227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.194
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:58:34.922230 30533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:58:34.922403 30533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:58:34.927520 30538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:34.927448 30541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:34.927462 30539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:34.927644 30533 server_base.cc:1034] running on GCE node
I20260929 10:58:34.927860 30533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:58:34.928135 30533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:58:34.929342 30533 hybrid_clock.cc:648] HybridClock initialized: now 1790679514929337 us; error 45 us; skew 500 ppm
Sep 29 10:58:34 dist-test-slave-8nrt krb5kdc[30292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679514, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:58:34.931816 30533 init.cc:377] Logged in from keytab as kudu/127.10.238.194@KRBTEST.COM (short username kudu)
I20260929 10:58:34.932257 30533 webserver.cc:460] Webserver started at http://127.10.238.194:37833/ using document root <none> and password file <none>
I20260929 10:58:34.932512 30533 fs_manager.cc:359] Metadata directory not provided
I20260929 10:58:34.932585 30533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:58:34.932742 30533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:58:34.933871 30533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/ts-1/data/instance:
uuid: "cdc3df9d28764becbc3a6025b3db2fb4"
format_stamp: "Formatted at 2026-09-29 10:58:34 on dist-test-slave-8nrt"
server_key: "176178a6cd75ba7af23490e85ad8cf7b"
server_key_iv: "53f50ea8b1739e683efdb896eefaa7b9"
server_key_version: "encryptionkey@0"
I20260929 10:58:34.934247 30533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/ts-1/wal/instance:
uuid: "cdc3df9d28764becbc3a6025b3db2fb4"
format_stamp: "Formatted at 2026-09-29 10:58:34 on dist-test-slave-8nrt"
server_key: "176178a6cd75ba7af23490e85ad8cf7b"
server_key_iv: "53f50ea8b1739e683efdb896eefaa7b9"
server_key_version: "encryptionkey@0"
I20260929 10:58:34.935572 30533 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 10:58:34.936316 30548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:34.936502 30533 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:58:34.936595 30533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/ts-1/wal
uuid: "cdc3df9d28764becbc3a6025b3db2fb4"
format_stamp: "Formatted at 2026-09-29 10:58:34 on dist-test-slave-8nrt"
server_key: "176178a6cd75ba7af23490e85ad8cf7b"
server_key_iv: "53f50ea8b1739e683efdb896eefaa7b9"
server_key_version: "encryptionkey@0"
I20260929 10:58:34.936677 30533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:58:34.952555 30533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:58:34.954484 30533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:58:34.954656 30533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:58:34.954905 30533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:58:34.955232 30533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:58:34.955287 30533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:34.955329 30533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:58:34.955358 30533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:34.961309 30533 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.194:34757
I20260929 10:58:34.961340 30661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.194:34757 every 8 connection(s)
I20260929 10:58:34.961666 30533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
Sep 29 10:58:34 dist-test-slave-8nrt krb5kdc[30292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679514, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.194@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:58:34.970170 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 30533
I20260929 10:58:34.970312 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/ts-1/wal/instance
I20260929 10:58:34.970592 11195 external_mini_cluster.cc:1428] Setting key 3d4b528ce75f9050d81ebac270f2e551
I20260929 10:58:34.971028 30378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:34.962405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:39969 (local address 127.10.238.254:41227)
0929 10:58:34.962481 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:34.962484 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:34.962667 (+ 183us) server_negotiation.cc:407] Connection header received
0929 10:58:34.963136 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:34.963137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:34.963147 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:34.963204 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:34.964091 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:34.964576 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:34.965633 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:34.965822 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:34.965875 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:34.967937 (+ 2062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:34.967939 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:34.967939 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:58:34.967945 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:34.969469 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:34.969831 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:34.969832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:34.969832 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:34.969883 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:34.970227 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:34.970228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:34.970228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:34.970422 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:34.970483 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:34.970844 (+ 361us) server_negotiation.cc:299] Negotiation successful
0929 10:58:34.970932 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 10:58:34.971267 30664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:34.962361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:41227 (local address 127.10.238.194:39969)
0929 10:58:34.962532 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:34.962573 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 10:58:34.963053 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:58:34.963262 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:58:34.963266 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:58:34.963817 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:58:34.964003 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:34.964007 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:34.964658 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:34.964658 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:34.965562 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:34.965564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:34.965897 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:34.965897 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:34.965995 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:34.966449 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:58:34.966457 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:58:34.967853 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:58:34.969539 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:58:34.969545 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:58:34.969547 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:58:34.969765 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:58:34.969962 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:58:34.969963 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:58:34.969963 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:58:34.970149 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:58:34.970495 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:58:34.970502 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:58:34.970679 (+ 177us) client_negotiation.cc:769] Sending connection context
0929 10:58:34.970794 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 10:58:34.970987 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":95,"threads_started":1}
I20260929 10:58:34.972090 30662 heartbeater.cc:347] Connected to a master server at 127.10.238.254:41227
I20260929 10:58:34.972223 30662 heartbeater.cc:464] Registering TS with master...
I20260929 10:58:34.972407 30662 heartbeater.cc:511] Master 127.10.238.254:41227 requested a full tablet report, sending...
I20260929 10:58:34.972810 30337 ts_manager.cc:194] Registered new tserver with Master: cdc3df9d28764becbc3a6025b3db2fb4 (127.10.238.194:34757)
I20260929 10:58:34.973199 30337 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.194@KRBTEST.COM'} at 127.10.238.194:39969
WARNING: no policy specified for kudu/127.10.238.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab.
Entry for principal kudu/127.10.238.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab.
I20260929 10:58:34.995464 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.195:0
--local_ip_for_outbound_sockets=127.10.238.195
--webserver_interface=127.10.238.195
--webserver_port=0
--tserver_master_addrs=127.10.238.254:41227
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab
--principal=kudu/127.10.238.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:40269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:58:35.077136 30667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:58:35.077435 30667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:58:35.077484 30667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:58:35.078779 30667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:58:35.078822 30667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:58:35.078876 30667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.195
I20260929 10:58:35.080476 30667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:40269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.195.keytab
--principal=kudu/127.10.238.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.238.195
--webserver_port=0
--tserver_master_addrs=127.10.238.254:41227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.195
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:58:35.080724 30667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:58:35.080958 30667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 10:58:35.086088 30667 server_base.cc:1034] running on GCE node
W20260929 10:58:35.086133 30673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:35.086122 30672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:35.086404 30675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:35.086650 30667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:58:35.086915 30667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:58:35.088064 30667 hybrid_clock.cc:648] HybridClock initialized: now 1790679515088052 us; error 30 us; skew 500 ppm
Sep 29 10:58:35 dist-test-slave-8nrt krb5kdc[30292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679515, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:58:35.090575 30667 init.cc:377] Logged in from keytab as kudu/127.10.238.195@KRBTEST.COM (short username kudu)
I20260929 10:58:35.091073 30667 webserver.cc:460] Webserver started at http://127.10.238.195:37067/ using document root <none> and password file <none>
I20260929 10:58:35.091295 30667 fs_manager.cc:359] Metadata directory not provided
I20260929 10:58:35.091339 30667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:58:35.091501 30667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:58:35.092553 30667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/ts-2/data/instance:
uuid: "d1b5627d393c44e58e8e33e33fb352cd"
format_stamp: "Formatted at 2026-09-29 10:58:35 on dist-test-slave-8nrt"
server_key: "ff26a0fde6f11ffb573cb52e775aaa8c"
server_key_iv: "294acd308d1e5b7194fccb0aa2c8303a"
server_key_version: "encryptionkey@0"
I20260929 10:58:35.092926 30667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/ts-2/wal/instance:
uuid: "d1b5627d393c44e58e8e33e33fb352cd"
format_stamp: "Formatted at 2026-09-29 10:58:35 on dist-test-slave-8nrt"
server_key: "ff26a0fde6f11ffb573cb52e775aaa8c"
server_key_iv: "294acd308d1e5b7194fccb0aa2c8303a"
server_key_version: "encryptionkey@0"
I20260929 10:58:35.094316 30667 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 10:58:35.095062 30682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:35.095353 30667 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 10:58:35.095458 30667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/ts-2/wal
uuid: "d1b5627d393c44e58e8e33e33fb352cd"
format_stamp: "Formatted at 2026-09-29 10:58:35 on dist-test-slave-8nrt"
server_key: "ff26a0fde6f11ffb573cb52e775aaa8c"
server_key_iv: "294acd308d1e5b7194fccb0aa2c8303a"
server_key_version: "encryptionkey@0"
I20260929 10:58:35.095541 30667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:58:35.120631 30667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:58:35.122709 30667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:58:35.122876 30667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:58:35.123127 30667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:58:35.123445 30667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:58:35.123500 30667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:35.123543 30667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:58:35.123564 30667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:35.130231 30667 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.195:42907
I20260929 10:58:35.130277 30795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.195:42907 every 8 connection(s)
I20260929 10:58:35.130595 30667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
Sep 29 10:58:35 dist-test-slave-8nrt krb5kdc[30292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679515, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.195@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:58:35.140012 30378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:35.131299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:51479 (local address 127.10.238.254:41227)
0929 10:58:35.131370 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:35.131372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:35.131576 (+ 204us) server_negotiation.cc:407] Connection header received
0929 10:58:35.132115 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:35.132117 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:35.132129 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:35.132179 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:35.133168 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:35.133736 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:35.134751 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:35.134952 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:35.135036 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:35.137250 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:35.137251 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:35.137251 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:58:35.137298 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:35.138716 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:35.139086 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:35.139086 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:35.139087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:35.139141 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:35.139426 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:35.139427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:35.139427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:35.139538 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:35.139591 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:35.139840 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 10:58:35.139943 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 10:58:35.140142 30798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:35.131271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:41227 (local address 127.10.238.195:51479)
0929 10:58:35.131454 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:35.131498 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 10:58:35.131997 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:58:35.132256 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:58:35.132260 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:58:35.132869 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:58:35.133067 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:35.133072 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:35.133841 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:35.133842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:35.134638 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:35.134640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:35.135047 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:35.135048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:35.135170 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:35.135629 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:58:35.135637 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:58:35.137157 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:58:35.138801 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:58:35.138806 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:58:35.138809 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:58:35.139022 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:58:35.139200 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:58:35.139200 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:58:35.139201 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:58:35.139339 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:58:35.139597 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:58:35.139602 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:58:35.139717 (+ 115us) client_negotiation.cc:769] Sending connection context
0929 10:58:35.139813 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 10:58:35.139931 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":111,"threads_started":1}
I20260929 10:58:35.140380 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 30667
I20260929 10:58:35.140492 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/ts-2/wal/instance
I20260929 10:58:35.140650 30796 heartbeater.cc:347] Connected to a master server at 127.10.238.254:41227
I20260929 10:58:35.140735 11195 external_mini_cluster.cc:1428] Setting key d50c8ad7ccdb35d17d169f045d7080a6
I20260929 10:58:35.140760 30796 heartbeater.cc:464] Registering TS with master...
I20260929 10:58:35.140955 30796 heartbeater.cc:511] Master 127.10.238.254:41227 requested a full tablet report, sending...
I20260929 10:58:35.141408 30336 ts_manager.cc:194] Registered new tserver with Master: d1b5627d393c44e58e8e33e33fb352cd (127.10.238.195:42907)
I20260929 10:58:35.141814 30336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.238.195@KRBTEST.COM'} at 127.10.238.195:51479
I20260929 10:58:35.152410 11195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:58:35.159228 30378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:35.153750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58756 (local address 127.10.238.254:41227)
0929 10:58:35.153823 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:35.153825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:35.153846 (+ 21us) server_negotiation.cc:407] Connection header received
0929 10:58:35.153893 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:35.153894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:35.153903 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:35.153941 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:35.154455 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:35.154919 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:35.155600 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:35.155777 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:35.155823 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:35.157000 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:35.157000 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:35.157001 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:58:35.157006 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:35.158298 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:35.158597 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:35.158598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:35.158598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:35.158646 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:35.158812 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:35.158813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:35.158813 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:35.158953 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:35.159026 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:35.159113 (+ 87us) server_negotiation.cc:299] Negotiation successful
0929 10:58:35.159167 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 10:58:35.161024 30336 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58756:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:58:35.162693 30336 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:58:35.169580 30809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:35.165176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55388 (local address 127.10.238.195:42907)
0929 10:58:35.165548 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:35.165552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:35.165562 (+ 10us) server_negotiation.cc:407] Connection header received
0929 10:58:35.165610 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:35.165612 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:35.165627 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:35.165892 (+ 265us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:58:35.165977 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:35.167143 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:35.168438 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:35.169118 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:35.169198 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:35.169289 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:35.169450 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 10:58:35.169494 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":107,"threads_started":1}
I20260929 10:58:35.169579 30808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:35.165086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.195:42907 (local address 127.0.0.1:55388)
0929 10:58:35.165289 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:35.165332 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 10:58:35.165402 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:58:35.165734 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:58:35.165737 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:58:35.165739 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:58:35.165808 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:35.165810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:35.167252 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:35.167253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:35.168346 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:35.168348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:35.169208 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:35.169209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:35.169328 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:35.169345 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 10:58:35.169424 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 10:58:35.169486 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":42,"thread_start_us":52,"threads_started":1}
I20260929 10:58:35.169579 30807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:35.165092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:33983 (local address 127.0.0.1:48266)
0929 10:58:35.165449 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:35.165462 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:58:35.165529 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:58:35.165863 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:58:35.165864 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:58:35.165866 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:58:35.165898 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:35.165899 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:35.167285 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:35.167286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:35.168232 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:35.168234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:35.169206 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:35.169208 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:35.169332 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:35.169344 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 10:58:35.169424 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 10:58:35.169486 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":6,"thread_start_us":99,"threads_started":1}
I20260929 10:58:35.169880 30810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:35.165237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48266 (local address 127.10.238.193:33983)
0929 10:58:35.165438 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:35.165444 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:35.165516 (+ 72us) server_negotiation.cc:407] Connection header received
0929 10:58:35.165602 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:35.165605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:35.165622 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:35.165701 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:58:35.166001 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:35.167143 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:35.168319 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:35.169118 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:35.169198 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:35.169279 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:35.169730 (+ 451us) server_negotiation.cc:299] Negotiation successful
0929 10:58:35.169791 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":115,"threads_started":1}
I20260929 10:58:35.170780 30812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:35.165599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.194:34757 (local address 127.0.0.1:56520)
0929 10:58:35.165723 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:35.165735 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:58:35.165793 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:58:35.166100 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:58:35.166101 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:58:35.166108 (+ 7us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:58:35.166144 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:35.166145 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:35.167010 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:35.167011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:35.168326 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:35.168329 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:35.170523 (+ 2194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:35.170524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:35.170587 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:35.170591 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 10:58:35.170662 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 10:58:35.170708 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":81,"threads_started":1}
I20260929 10:58:35.170682 30462 tablet_service.cc:1467] Processing CreateTablet for tablet f95570d417a646359f8a1d5050e248d6 (DEFAULT_TABLE table=test-table [id=db989621b26d4d57baa53cd724144d27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:58:35.170840 30811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:35.165441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56520 (local address 127.10.238.194:34757)
0929 10:58:35.165775 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:35.165780 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:35.165791 (+ 11us) server_negotiation.cc:407] Connection header received
0929 10:58:35.165929 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:35.165931 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:35.165948 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:35.166021 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:58:35.166218 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:35.166906 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:35.169324 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:35.170362 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:35.170435 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:35.170488 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:35.170700 (+ 212us) server_negotiation.cc:299] Negotiation successful
0929 10:58:35.170764 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":82,"threads_started":1}
I20260929 10:58:35.170790 30730 tablet_service.cc:1467] Processing CreateTablet for tablet f95570d417a646359f8a1d5050e248d6 (DEFAULT_TABLE table=test-table [id=db989621b26d4d57baa53cd724144d27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:58:35.171043 30462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f95570d417a646359f8a1d5050e248d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:58:35.171144 30730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f95570d417a646359f8a1d5050e248d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:58:35.171821 30596 tablet_service.cc:1467] Processing CreateTablet for tablet f95570d417a646359f8a1d5050e248d6 (DEFAULT_TABLE table=test-table [id=db989621b26d4d57baa53cd724144d27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:58:35.172119 30596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f95570d417a646359f8a1d5050e248d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:58:35.174057 30813 tablet_bootstrap.cc:492] T f95570d417a646359f8a1d5050e248d6 P d1b5627d393c44e58e8e33e33fb352cd: Bootstrap starting.
I20260929 10:58:35.174502 30814 tablet_bootstrap.cc:492] T f95570d417a646359f8a1d5050e248d6 P 5afb1c5268df4855b6941d8665d931a1: Bootstrap starting.
I20260929 10:58:35.174875 30813 tablet_bootstrap.cc:654] T f95570d417a646359f8a1d5050e248d6 P d1b5627d393c44e58e8e33e33fb352cd: Neither blocks nor log segments found. Creating new log.
I20260929 10:58:35.175164 30814 tablet_bootstrap.cc:654] T f95570d417a646359f8a1d5050e248d6 P 5afb1c5268df4855b6941d8665d931a1: Neither blocks nor log segments found. Creating new log.
I20260929 10:58:35.175372 30815 tablet_bootstrap.cc:492] T f95570d417a646359f8a1d5050e248d6 P cdc3df9d28764becbc3a6025b3db2fb4: Bootstrap starting.
I20260929 10:58:35.175693 30813 log.cc:824] T f95570d417a646359f8a1d5050e248d6 P d1b5627d393c44e58e8e33e33fb352cd: Log is configured to *not* fsync() on all Append() calls
I20260929 10:58:35.175740 30814 log.cc:824] T f95570d417a646359f8a1d5050e248d6 P 5afb1c5268df4855b6941d8665d931a1: Log is configured to *not* fsync() on all Append() calls
I20260929 10:58:35.176019 30815 tablet_bootstrap.cc:654] T f95570d417a646359f8a1d5050e248d6 P cdc3df9d28764becbc3a6025b3db2fb4: Neither blocks nor log segments found. Creating new log.
I20260929 10:58:35.176489 30814 tablet_bootstrap.cc:492] T f95570d417a646359f8a1d5050e248d6 P 5afb1c5268df4855b6941d8665d931a1: No bootstrap required, opened a new log
I20260929 10:58:35.176501 30813 tablet_bootstrap.cc:492] T f95570d417a646359f8a1d5050e248d6 P d1b5627d393c44e58e8e33e33fb352cd: No bootstrap required, opened a new log
I20260929 10:58:35.176514 30815 log.cc:824] T f95570d417a646359f8a1d5050e248d6 P cdc3df9d28764becbc3a6025b3db2fb4: Log is configured to *not* fsync() on all Append() calls
I20260929 10:58:35.176561 30813 ts_tablet_manager.cc:1397] T f95570d417a646359f8a1d5050e248d6 P d1b5627d393c44e58e8e33e33fb352cd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 10:58:35.176632 30814 ts_tablet_manager.cc:1397] T f95570d417a646359f8a1d5050e248d6 P 5afb1c5268df4855b6941d8665d931a1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 10:58:35.177212 30815 tablet_bootstrap.cc:492] T f95570d417a646359f8a1d5050e248d6 P cdc3df9d28764becbc3a6025b3db2fb4: No bootstrap required, opened a new log
I20260929 10:58:35.177318 30815 ts_tablet_manager.cc:1397] T f95570d417a646359f8a1d5050e248d6 P cdc3df9d28764becbc3a6025b3db2fb4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 10:58:35.178609 30813 raft_consensus.cc:348] T f95570d417a646359f8a1d5050e248d6 P d1b5627d393c44e58e8e33e33fb352cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b5627d393c44e58e8e33e33fb352cd" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 42907 } } peers { permanent_uuid: "5afb1c5268df4855b6941d8665d931a1" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 33983 } } peers { permanent_uuid: "cdc3df9d28764becbc3a6025b3db2fb4" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 34757 } }
I20260929 10:58:35.178598 30814 raft_consensus.cc:348] T f95570d417a646359f8a1d5050e248d6 P 5afb1c5268df4855b6941d8665d931a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b5627d393c44e58e8e33e33fb352cd" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 42907 } } peers { permanent_uuid: "5afb1c5268df4855b6941d8665d931a1" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 33983 } } peers { permanent_uuid: "cdc3df9d28764becbc3a6025b3db2fb4" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 34757 } }
I20260929 10:58:35.178779 30813 raft_consensus.cc:374] T f95570d417a646359f8a1d5050e248d6 P d1b5627d393c44e58e8e33e33fb352cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:58:35.178793 30814 raft_consensus.cc:374] T f95570d417a646359f8a1d5050e248d6 P 5afb1c5268df4855b6941d8665d931a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:58:35.178825 30813 raft_consensus.cc:729] T f95570d417a646359f8a1d5050e248d6 P d1b5627d393c44e58e8e33e33fb352cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1b5627d393c44e58e8e33e33fb352cd, State: Initialized, Role: FOLLOWER
I20260929 10:58:35.178828 30814 raft_consensus.cc:729] T f95570d417a646359f8a1d5050e248d6 P 5afb1c5268df4855b6941d8665d931a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5afb1c5268df4855b6941d8665d931a1, State: Initialized, Role: FOLLOWER
I20260929 10:58:35.178784 30815 raft_consensus.cc:348] T f95570d417a646359f8a1d5050e248d6 P cdc3df9d28764becbc3a6025b3db2fb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b5627d393c44e58e8e33e33fb352cd" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 42907 } } peers { permanent_uuid: "5afb1c5268df4855b6941d8665d931a1" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 33983 } } peers { permanent_uuid: "cdc3df9d28764becbc3a6025b3db2fb4" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 34757 } }
I20260929 10:58:35.178968 30815 raft_consensus.cc:374] T f95570d417a646359f8a1d5050e248d6 P cdc3df9d28764becbc3a6025b3db2fb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:58:35.178967 30813 consensus_queue.cc:260] T f95570d417a646359f8a1d5050e248d6 P d1b5627d393c44e58e8e33e33fb352cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b5627d393c44e58e8e33e33fb352cd" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 42907 } } peers { permanent_uuid: "5afb1c5268df4855b6941d8665d931a1" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 33983 } } peers { permanent_uuid: "cdc3df9d28764becbc3a6025b3db2fb4" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 34757 } }
I20260929 10:58:35.178977 30814 consensus_queue.cc:260] T f95570d417a646359f8a1d5050e248d6 P 5afb1c5268df4855b6941d8665d931a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b5627d393c44e58e8e33e33fb352cd" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 42907 } } peers { permanent_uuid: "5afb1c5268df4855b6941d8665d931a1" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 33983 } } peers { permanent_uuid: "cdc3df9d28764becbc3a6025b3db2fb4" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 34757 } }
I20260929 10:58:35.179014 30815 raft_consensus.cc:729] T f95570d417a646359f8a1d5050e248d6 P cdc3df9d28764becbc3a6025b3db2fb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdc3df9d28764becbc3a6025b3db2fb4, State: Initialized, Role: FOLLOWER
I20260929 10:58:35.179147 30815 consensus_queue.cc:260] T f95570d417a646359f8a1d5050e248d6 P cdc3df9d28764becbc3a6025b3db2fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b5627d393c44e58e8e33e33fb352cd" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 42907 } } peers { permanent_uuid: "5afb1c5268df4855b6941d8665d931a1" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 33983 } } peers { permanent_uuid: "cdc3df9d28764becbc3a6025b3db2fb4" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 34757 } }
I20260929 10:58:35.179263 30814 ts_tablet_manager.cc:1428] T f95570d417a646359f8a1d5050e248d6 P 5afb1c5268df4855b6941d8665d931a1: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260929 10:58:35.179263 30813 ts_tablet_manager.cc:1428] T f95570d417a646359f8a1d5050e248d6 P d1b5627d393c44e58e8e33e33fb352cd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 10:58:35.179381 30528 heartbeater.cc:503] Master 127.10.238.254:41227 was elected leader, sending a full tablet report...
I20260929 10:58:35.179427 30815 ts_tablet_manager.cc:1428] T f95570d417a646359f8a1d5050e248d6 P cdc3df9d28764becbc3a6025b3db2fb4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 10:58:35.179453 30796 heartbeater.cc:503] Master 127.10.238.254:41227 was elected leader, sending a full tablet report...
I20260929 10:58:35.179534 30662 heartbeater.cc:503] Master 127.10.238.254:41227 was elected leader, sending a full tablet report...
W20260929 10:58:35.212308 30663 tablet.cc:2369] T f95570d417a646359f8a1d5050e248d6 P cdc3df9d28764becbc3a6025b3db2fb4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 10:58:35.299592 30529 tablet.cc:2369] T f95570d417a646359f8a1d5050e248d6 P 5afb1c5268df4855b6941d8665d931a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 10:58:35.381150 30797 tablet.cc:2369] T f95570d417a646359f8a1d5050e248d6 P d1b5627d393c44e58e8e33e33fb352cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:58:35.536067 30821 raft_consensus.cc:482] T f95570d417a646359f8a1d5050e248d6 P cdc3df9d28764becbc3a6025b3db2fb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:58:35.536238 30821 raft_consensus.cc:504] T f95570d417a646359f8a1d5050e248d6 P cdc3df9d28764becbc3a6025b3db2fb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b5627d393c44e58e8e33e33fb352cd" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 42907 } } peers { permanent_uuid: "5afb1c5268df4855b6941d8665d931a1" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 33983 } } peers { permanent_uuid: "cdc3df9d28764becbc3a6025b3db2fb4" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 34757 } }
I20260929 10:58:35.536795 30821 leader_election.cc:288] T f95570d417a646359f8a1d5050e248d6 P cdc3df9d28764becbc3a6025b3db2fb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1b5627d393c44e58e8e33e33fb352cd (127.10.238.195:42907), 5afb1c5268df4855b6941d8665d931a1 (127.10.238.193:33983)
I20260929 10:58:35.541162 30810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:35.537091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:34683 (local address 127.10.238.193:33983)
0929 10:58:35.537145 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:35.537148 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:35.537197 (+ 49us) server_negotiation.cc:407] Connection header received
0929 10:58:35.537328 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:35.537330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:35.537339 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:35.537415 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:58:35.537662 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:35.538461 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:35.540134 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:35.540780 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:35.540857 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:35.540868 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:35.541042 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 10:58:35.541087 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260929 10:58:35.541163 30809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:35.537135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:57067 (local address 127.10.238.195:42907)
0929 10:58:35.537181 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:35.537184 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:35.537197 (+ 13us) server_negotiation.cc:407] Connection header received
0929 10:58:35.537320 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:35.537321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:35.537332 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:35.537415 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:58:35.537626 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:35.538464 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:35.540152 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:35.540764 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:35.540858 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:35.540868 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:35.541042 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 10:58:35.541098 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 10:58:35.541188 30822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:35.537000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:33983 (local address 127.10.238.194:34683)
0929 10:58:35.537129 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:35.537142 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:58:35.537199 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:58:35.537431 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:58:35.537431 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:58:35.537435 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:58:35.537539 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:35.537541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:35.538560 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:35.538561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:35.539998 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:35.540001 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:35.540866 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:35.540867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:35.540948 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:35.540953 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 10:58:35.541016 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 10:58:35.541076 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":108,"thread_start_us":60,"threads_started":1}
I20260929 10:58:35.541188 30823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:35.537000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.195:42907 (local address 127.10.238.194:57067)
0929 10:58:35.537129 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:35.537142 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:58:35.537199 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:58:35.537430 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:58:35.537431 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:58:35.537434 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:58:35.537539 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:35.537543 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:35.538560 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:35.538561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:35.539998 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:35.540001 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:35.540864 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:35.540866 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:35.540948 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:35.540954 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 10:58:35.541016 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 10:58:35.541076 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":123,"thread_start_us":61,"threads_started":1}
I20260929 10:58:35.541590 30482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95570d417a646359f8a1d5050e248d6" candidate_uuid: "cdc3df9d28764becbc3a6025b3db2fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5afb1c5268df4855b6941d8665d931a1" is_pre_election: true
I20260929 10:58:35.541647 30750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95570d417a646359f8a1d5050e248d6" candidate_uuid: "cdc3df9d28764becbc3a6025b3db2fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1b5627d393c44e58e8e33e33fb352cd" is_pre_election: true
I20260929 10:58:35.541771 30482 raft_consensus.cc:2454] T f95570d417a646359f8a1d5050e248d6 P 5afb1c5268df4855b6941d8665d931a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdc3df9d28764becbc3a6025b3db2fb4 in term 0.
I20260929 10:58:35.541771 30750 raft_consensus.cc:2454] T f95570d417a646359f8a1d5050e248d6 P d1b5627d393c44e58e8e33e33fb352cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdc3df9d28764becbc3a6025b3db2fb4 in term 0.
I20260929 10:58:35.542027 30551 leader_election.cc:302] T f95570d417a646359f8a1d5050e248d6 P cdc3df9d28764becbc3a6025b3db2fb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5afb1c5268df4855b6941d8665d931a1, cdc3df9d28764becbc3a6025b3db2fb4; no voters:
I20260929 10:58:35.542176 30821 raft_consensus.cc:2781] T f95570d417a646359f8a1d5050e248d6 P cdc3df9d28764becbc3a6025b3db2fb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:58:35.542268 30821 raft_consensus.cc:482] T f95570d417a646359f8a1d5050e248d6 P cdc3df9d28764becbc3a6025b3db2fb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:58:35.542290 30821 raft_consensus.cc:3037] T f95570d417a646359f8a1d5050e248d6 P cdc3df9d28764becbc3a6025b3db2fb4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:58:35.543100 30821 raft_consensus.cc:504] T f95570d417a646359f8a1d5050e248d6 P cdc3df9d28764becbc3a6025b3db2fb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b5627d393c44e58e8e33e33fb352cd" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 42907 } } peers { permanent_uuid: "5afb1c5268df4855b6941d8665d931a1" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 33983 } } peers { permanent_uuid: "cdc3df9d28764becbc3a6025b3db2fb4" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 34757 } }
I20260929 10:58:35.543264 30821 leader_election.cc:288] T f95570d417a646359f8a1d5050e248d6 P cdc3df9d28764becbc3a6025b3db2fb4 [CANDIDATE]: Term 1 election: Requested vote from peers d1b5627d393c44e58e8e33e33fb352cd (127.10.238.195:42907), 5afb1c5268df4855b6941d8665d931a1 (127.10.238.193:33983)
I20260929 10:58:35.543458 30482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95570d417a646359f8a1d5050e248d6" candidate_uuid: "cdc3df9d28764becbc3a6025b3db2fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5afb1c5268df4855b6941d8665d931a1"
I20260929 10:58:35.543525 30750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95570d417a646359f8a1d5050e248d6" candidate_uuid: "cdc3df9d28764becbc3a6025b3db2fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1b5627d393c44e58e8e33e33fb352cd"
I20260929 10:58:35.543565 30482 raft_consensus.cc:3037] T f95570d417a646359f8a1d5050e248d6 P 5afb1c5268df4855b6941d8665d931a1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:58:35.543592 30750 raft_consensus.cc:3037] T f95570d417a646359f8a1d5050e248d6 P d1b5627d393c44e58e8e33e33fb352cd [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:58:35.544519 30482 raft_consensus.cc:2454] T f95570d417a646359f8a1d5050e248d6 P 5afb1c5268df4855b6941d8665d931a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdc3df9d28764becbc3a6025b3db2fb4 in term 1.
I20260929 10:58:35.544519 30750 raft_consensus.cc:2454] T f95570d417a646359f8a1d5050e248d6 P d1b5627d393c44e58e8e33e33fb352cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdc3df9d28764becbc3a6025b3db2fb4 in term 1.
I20260929 10:58:35.544783 30549 leader_election.cc:302] T f95570d417a646359f8a1d5050e248d6 P cdc3df9d28764becbc3a6025b3db2fb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cdc3df9d28764becbc3a6025b3db2fb4, d1b5627d393c44e58e8e33e33fb352cd; no voters:
I20260929 10:58:35.544883 30821 raft_consensus.cc:2781] T f95570d417a646359f8a1d5050e248d6 P cdc3df9d28764becbc3a6025b3db2fb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:58:35.545089 30821 raft_consensus.cc:686] T f95570d417a646359f8a1d5050e248d6 P cdc3df9d28764becbc3a6025b3db2fb4 [term 1 LEADER]: Becoming Leader. State: Replica: cdc3df9d28764becbc3a6025b3db2fb4, State: Running, Role: LEADER
I20260929 10:58:35.545194 30821 consensus_queue.cc:237] T f95570d417a646359f8a1d5050e248d6 P cdc3df9d28764becbc3a6025b3db2fb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b5627d393c44e58e8e33e33fb352cd" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 42907 } } peers { permanent_uuid: "5afb1c5268df4855b6941d8665d931a1" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 33983 } } peers { permanent_uuid: "cdc3df9d28764becbc3a6025b3db2fb4" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 34757 } }
I20260929 10:58:35.546089 30337 catalog_manager.cc:5495] T f95570d417a646359f8a1d5050e248d6 P cdc3df9d28764becbc3a6025b3db2fb4 reported cstate change: term changed from 0 to 1, leader changed from <none> to cdc3df9d28764becbc3a6025b3db2fb4 (127.10.238.194). New cstate: current_term: 1 leader_uuid: "cdc3df9d28764becbc3a6025b3db2fb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b5627d393c44e58e8e33e33fb352cd" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 42907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5afb1c5268df4855b6941d8665d931a1" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 33983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdc3df9d28764becbc3a6025b3db2fb4" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 34757 } health_report { overall_health: HEALTHY } } }
I20260929 10:58:35.608950 30811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:35.606384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52702 (local address 127.10.238.194:34757)
0929 10:58:35.606462 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:35.606464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:35.606474 (+ 10us) server_negotiation.cc:407] Connection header received
0929 10:58:35.606523 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:35.606524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:35.606543 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:35.606623 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:58:35.606835 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:35.607341 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:35.608135 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:35.608337 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:35.608415 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:35.608609 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:58:35.608716 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:58:35.608814 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:35.608834 (+ 20us) server_negotiation.cc:299] Negotiation successful
0929 10:58:35.608858 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 10:58:35.610828 30482 raft_consensus.cc:1261] T f95570d417a646359f8a1d5050e248d6 P 5afb1c5268df4855b6941d8665d931a1 [term 1 FOLLOWER]: Refusing update from remote peer cdc3df9d28764becbc3a6025b3db2fb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:58:35.610847 30750 raft_consensus.cc:1261] T f95570d417a646359f8a1d5050e248d6 P d1b5627d393c44e58e8e33e33fb352cd [term 1 FOLLOWER]: Refusing update from remote peer cdc3df9d28764becbc3a6025b3db2fb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:58:35.611186 30821 consensus_queue.cc:1035] T f95570d417a646359f8a1d5050e248d6 P cdc3df9d28764becbc3a6025b3db2fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5afb1c5268df4855b6941d8665d931a1" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 33983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:58:35.611346 30824 consensus_queue.cc:1035] T f95570d417a646359f8a1d5050e248d6 P cdc3df9d28764becbc3a6025b3db2fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1b5627d393c44e58e8e33e33fb352cd" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 42907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:58:35.615012 30829 mvcc.cc:205] Tried to move back new op lower bound from 7334623295939182592 to 7334623295673892864. Current Snapshot: MvccSnapshot[applied={T|T < 7334623295939182592 or (T in {7334623295939182592})}]
I20260929 10:58:35.615151 30828 mvcc.cc:205] Tried to move back new op lower bound from 7334623295939182592 to 7334623295673892864. Current Snapshot: MvccSnapshot[applied={T|T < 7334623295939182592 or (T in {7334623295939182592})}]
I20260929 10:58:35.615998 30335 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:58:35.616078 30335 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:58:35.616878 30335 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2ce90d4b5e6d400181ab6d655379a2a4: Sending DeleteTablet for 3 replicas of tablet f95570d417a646359f8a1d5050e248d6
I20260929 10:58:35.617223 30462 tablet_service.cc:1514] Processing DeleteTablet for tablet f95570d417a646359f8a1d5050e248d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:58:35 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:48266
I20260929 10:58:35.617283 30730 tablet_service.cc:1514] Processing DeleteTablet for tablet f95570d417a646359f8a1d5050e248d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:58:35 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:55388
I20260929 10:58:35.617455 30836 tablet_replica.cc:330] T f95570d417a646359f8a1d5050e248d6 P 5afb1c5268df4855b6941d8665d931a1: stopping tablet replica
I20260929 10:58:35.617547 30836 raft_consensus.cc:2229] T f95570d417a646359f8a1d5050e248d6 P 5afb1c5268df4855b6941d8665d931a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:58:35.617560 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 30399
I20260929 10:58:35.617642 30836 raft_consensus.cc:2258] T f95570d417a646359f8a1d5050e248d6 P 5afb1c5268df4855b6941d8665d931a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:58:35.620196 30837 tablet_replica.cc:330] T f95570d417a646359f8a1d5050e248d6 P d1b5627d393c44e58e8e33e33fb352cd: stopping tablet replica
I20260929 10:58:35.620177 30596 tablet_service.cc:1514] Processing DeleteTablet for tablet f95570d417a646359f8a1d5050e248d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:58:35 UTC) from {username='kudu', principal='kudu/127.10.238.254@KRBTEST.COM'} at 127.0.0.1:56520
I20260929 10:58:35.620328 30837 raft_consensus.cc:2229] T f95570d417a646359f8a1d5050e248d6 P d1b5627d393c44e58e8e33e33fb352cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:58:35.620432 30837 raft_consensus.cc:2258] T f95570d417a646359f8a1d5050e248d6 P d1b5627d393c44e58e8e33e33fb352cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:58:35.620991 30837 ts_tablet_manager.cc:1905] T f95570d417a646359f8a1d5050e248d6 P d1b5627d393c44e58e8e33e33fb352cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:58:35.622901 30837 ts_tablet_manager.cc:1918] T f95570d417a646359f8a1d5050e248d6 P d1b5627d393c44e58e8e33e33fb352cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:58:35.622977 30837 log.cc:1196] T f95570d417a646359f8a1d5050e248d6 P d1b5627d393c44e58e8e33e33fb352cd: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790679274510869-11195-0/minicluster-data/ts-2/wal/wals/f95570d417a646359f8a1d5050e248d6
I20260929 10:58:35.623260 30837 ts_tablet_manager.cc:1939] T f95570d417a646359f8a1d5050e248d6 P d1b5627d393c44e58e8e33e33fb352cd: Deleting consensus metadata
I20260929 10:58:35.623672 30321 catalog_manager.cc:4841] TS d1b5627d393c44e58e8e33e33fb352cd (127.10.238.195:42907): tablet f95570d417a646359f8a1d5050e248d6 (table test-table [id=db989621b26d4d57baa53cd724144d27]) successfully deleted
W20260929 10:58:35.624414 30323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.238.193:33983 (error 108)
I20260929 10:58:35.624606 30838 tablet_replica.cc:330] T f95570d417a646359f8a1d5050e248d6 P cdc3df9d28764becbc3a6025b3db2fb4: stopping tablet replica
I20260929 10:58:35.624712 30838 raft_consensus.cc:2229] T f95570d417a646359f8a1d5050e248d6 P cdc3df9d28764becbc3a6025b3db2fb4 [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:58:35.624810 30838 raft_consensus.cc:2258] T f95570d417a646359f8a1d5050e248d6 P cdc3df9d28764becbc3a6025b3db2fb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:58:35.624960 30812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:35.624822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:33983 (local address 127.0.0.1:47792)
0929 10:58:35.624887 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:35.624929 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.193:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 10:58:35.624992 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 30533
W20260929 10:58:35.625823 30323 catalog_manager.cc:4584] TS 5afb1c5268df4855b6941d8665d931a1 (127.10.238.193:33983): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f95570d417a646359f8a1d5050e248d6: Network error: Client connection negotiation failed: client connection to 127.10.238.193:33983: connect: Connection refused (error 111)
I20260929 10:58:35.630793 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 30667
I20260929 10:58:35.630836 30812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:35.630684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.194:34757 (local address 127.0.0.1:52714)
0929 10:58:35.630782 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:35.630813 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.194:34757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260929 10:58:35.635522 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 30305
2026-09-29T10:58:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2291 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T10:58:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:58:35Z Disabled control of system clock
I20260929 10:58:35.663077 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:35397
--webserver_interface=127.10.238.254
--webserver_port=0
--builtin_ntp_servers=127.10.238.212:39411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:35397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 10:58:35.742703 30843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:58:35.743034 30843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:58:35.743070 30843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:58:35.744765 30843 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:58:35.744808 30843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:58:35.744822 30843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:58:35.744834 30843 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:58:35.744846 30843 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:58:35.746153 30843 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:39411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:35397
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:35397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:58:35.746358 30843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:58:35.746588 30843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:58:35.752264 30851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:35.752290 30848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:35.752485 30843 server_base.cc:1034] running on GCE node
W20260929 10:58:35.752319 30849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:35.752769 30843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:58:35.753055 30843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:58:35.754221 30843 hybrid_clock.cc:648] HybridClock initialized: now 1790679515754190 us; error 47 us; skew 500 ppm
I20260929 10:58:35.755506 30843 webserver.cc:460] Webserver started at http://127.10.238.254:38725/ using document root <none> and password file <none>
I20260929 10:58:35.755780 30843 fs_manager.cc:359] Metadata directory not provided
I20260929 10:58:35.755856 30843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:58:35.756007 30843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:58:35.757077 30843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/master-0/data/instance:
uuid: "f19af86671f2492f81f5f729d739b24a"
format_stamp: "Formatted at 2026-09-29 10:58:35 on dist-test-slave-8nrt"
server_key: "c541acfcc4aa9af88b7a2a0d80482b42"
server_key_iv: "f3b406fd1df704055a83be86d5a1cb42"
server_key_version: "encryptionkey@0"
I20260929 10:58:35.757483 30843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/master-0/wal/instance:
uuid: "f19af86671f2492f81f5f729d739b24a"
format_stamp: "Formatted at 2026-09-29 10:58:35 on dist-test-slave-8nrt"
server_key: "c541acfcc4aa9af88b7a2a0d80482b42"
server_key_iv: "f3b406fd1df704055a83be86d5a1cb42"
server_key_version: "encryptionkey@0"
I20260929 10:58:35.758749 30843 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 10:58:35.759534 30857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:35.759713 30843 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 10:58:35.759812 30843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "f19af86671f2492f81f5f729d739b24a"
format_stamp: "Formatted at 2026-09-29 10:58:35 on dist-test-slave-8nrt"
server_key: "c541acfcc4aa9af88b7a2a0d80482b42"
server_key_iv: "f3b406fd1df704055a83be86d5a1cb42"
server_key_version: "encryptionkey@0"
I20260929 10:58:35.759895 30843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:58:35.768268 30843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:58:35.768561 30843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:58:35.768709 30843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:58:35.772779 30843 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.254:35397
I20260929 10:58:35.772811 30909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.254:35397 every 8 connection(s)
I20260929 10:58:35.773145 30843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:58:35.774121 30910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:58:35.776376 30910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f19af86671f2492f81f5f729d739b24a: Bootstrap starting.
I20260929 10:58:35.776957 30910 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f19af86671f2492f81f5f729d739b24a: Neither blocks nor log segments found. Creating new log.
I20260929 10:58:35.777184 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 30843
I20260929 10:58:35.777323 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/master-0/wal/instance
I20260929 10:58:35.777319 30910 log.cc:824] T 00000000000000000000000000000000 P f19af86671f2492f81f5f729d739b24a: Log is configured to *not* fsync() on all Append() calls
I20260929 10:58:35.777534 11195 external_mini_cluster.cc:1428] Setting key ef6b86d6ee80b0d2a1500027aa620168
I20260929 10:58:35.778110 30910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f19af86671f2492f81f5f729d739b24a: No bootstrap required, opened a new log
I20260929 10:58:35.780812 30910 raft_consensus.cc:348] T 00000000000000000000000000000000 P f19af86671f2492f81f5f729d739b24a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f19af86671f2492f81f5f729d739b24a" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 35397 } }
I20260929 10:58:35.780973 30910 raft_consensus.cc:374] T 00000000000000000000000000000000 P f19af86671f2492f81f5f729d739b24a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:58:35.781002 30910 raft_consensus.cc:729] T 00000000000000000000000000000000 P f19af86671f2492f81f5f729d739b24a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19af86671f2492f81f5f729d739b24a, State: Initialized, Role: FOLLOWER
I20260929 10:58:35.781106 30910 consensus_queue.cc:260] T 00000000000000000000000000000000 P f19af86671f2492f81f5f729d739b24a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f19af86671f2492f81f5f729d739b24a" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 35397 } }
I20260929 10:58:35.781172 30910 raft_consensus.cc:388] T 00000000000000000000000000000000 P f19af86671f2492f81f5f729d739b24a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:58:35.781203 30910 raft_consensus.cc:482] T 00000000000000000000000000000000 P f19af86671f2492f81f5f729d739b24a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:58:35.781278 30910 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f19af86671f2492f81f5f729d739b24a [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:58:35.781814 30910 raft_consensus.cc:504] T 00000000000000000000000000000000 P f19af86671f2492f81f5f729d739b24a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f19af86671f2492f81f5f729d739b24a" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 35397 } }
I20260929 10:58:35.781937 30910 leader_election.cc:302] T 00000000000000000000000000000000 P f19af86671f2492f81f5f729d739b24a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f19af86671f2492f81f5f729d739b24a; no voters:
I20260929 10:58:35.782115 30910 leader_election.cc:288] T 00000000000000000000000000000000 P f19af86671f2492f81f5f729d739b24a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:58:35.782142 30915 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f19af86671f2492f81f5f729d739b24a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:58:35.782253 30915 raft_consensus.cc:686] T 00000000000000000000000000000000 P f19af86671f2492f81f5f729d739b24a [term 1 LEADER]: Becoming Leader. State: Replica: f19af86671f2492f81f5f729d739b24a, State: Running, Role: LEADER
I20260929 10:58:35.782331 30915 consensus_queue.cc:237] T 00000000000000000000000000000000 P f19af86671f2492f81f5f729d739b24a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f19af86671f2492f81f5f729d739b24a" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 35397 } }
I20260929 10:58:35.782374 30910 sys_catalog.cc:564] T 00000000000000000000000000000000 P f19af86671f2492f81f5f729d739b24a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:58:35.782649 30915 sys_catalog.cc:455] T 00000000000000000000000000000000 P f19af86671f2492f81f5f729d739b24a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f19af86671f2492f81f5f729d739b24a. Latest consensus state: current_term: 1 leader_uuid: "f19af86671f2492f81f5f729d739b24a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f19af86671f2492f81f5f729d739b24a" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 35397 } } }
I20260929 10:58:35.782728 30915 sys_catalog.cc:458] T 00000000000000000000000000000000 P f19af86671f2492f81f5f729d739b24a [sys.catalog]: This master's current role is: LEADER
I20260929 10:58:35.782660 30916 sys_catalog.cc:455] T 00000000000000000000000000000000 P f19af86671f2492f81f5f729d739b24a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f19af86671f2492f81f5f729d739b24a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f19af86671f2492f81f5f729d739b24a" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 35397 } } }
I20260929 10:58:35.783226 30916 sys_catalog.cc:458] T 00000000000000000000000000000000 P f19af86671f2492f81f5f729d739b24a [sys.catalog]: This master's current role is: LEADER
I20260929 10:58:35.783535 30920 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:58:35.784017 30920 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:58:35.784420 30914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:35.778041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39828 (local address 127.10.238.254:35397)
0929 10:58:35.778261 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:35.778271 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:35.778311 (+ 40us) server_negotiation.cc:407] Connection header received
0929 10:58:35.779355 (+ 1044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:35.779364 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:35.779439 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:35.779590 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:35.779740 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:35.781166 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:35.782086 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:35.782732 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:35.782850 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:35.783589 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:35.783592 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:35.783593 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:58:35.783809 (+ 216us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:35.783848 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:58:35.783862 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:35.783927 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:35.784215 (+ 288us) server_negotiation.cc:299] Negotiation successful
0929 10:58:35.784278 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":82,"threads_started":1}
I20260929 10:58:35.786270 30920 catalog_manager.cc:1347] Generated new cluster ID: 3ae4a53bd3d2410b973a289d35914849
I20260929 10:58:35.786326 30920 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:58:35.803323 30920 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:58:35.804018 30920 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:58:35.813424 30920 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f19af86671f2492f81f5f729d739b24a: Generated new TSK 0
I20260929 10:58:35.813608 30920 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:58:35.815101 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.193:0
--local_ip_for_outbound_sockets=127.10.238.193
--webserver_interface=127.10.238.193
--webserver_port=0
--tserver_master_addrs=127.10.238.254:35397
--builtin_ntp_servers=127.10.238.212:39411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 10:58:35.913094 30934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:58:35.913312 30934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:58:35.913383 30934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:58:35.914526 30934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:58:35.914595 30934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:58:35.914635 30934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.193
I20260929 10:58:35.915968 30934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:39411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.238.193
--webserver_port=0
--tserver_master_addrs=127.10.238.254:35397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.193
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:58:35.916231 30934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:58:35.916541 30934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:58:35.921626 30939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:35.921690 30940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:35.921801 30934 server_base.cc:1034] running on GCE node
W20260929 10:58:35.921914 30942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:35.922109 30934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:58:35.922557 30934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:58:35.923748 30934 hybrid_clock.cc:648] HybridClock initialized: now 1790679515923727 us; error 39 us; skew 500 ppm
I20260929 10:58:35.924980 30934 webserver.cc:460] Webserver started at http://127.10.238.193:36441/ using document root <none> and password file <none>
I20260929 10:58:35.925237 30934 fs_manager.cc:359] Metadata directory not provided
I20260929 10:58:35.925334 30934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:58:35.925469 30934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:58:35.926597 30934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/ts-0/data/instance:
uuid: "70baca43719d4f988294386be9779ab3"
format_stamp: "Formatted at 2026-09-29 10:58:35 on dist-test-slave-8nrt"
server_key: "402fbac42776a7701f9ec3505fbc2134"
server_key_iv: "f25b45ba080a4355534a5d62df5fcf48"
server_key_version: "encryptionkey@0"
I20260929 10:58:35.926980 30934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/ts-0/wal/instance:
uuid: "70baca43719d4f988294386be9779ab3"
format_stamp: "Formatted at 2026-09-29 10:58:35 on dist-test-slave-8nrt"
server_key: "402fbac42776a7701f9ec3505fbc2134"
server_key_iv: "f25b45ba080a4355534a5d62df5fcf48"
server_key_version: "encryptionkey@0"
I20260929 10:58:35.928339 30934 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 10:58:35.929203 30948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:35.929417 30934 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:58:35.929522 30934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/ts-0/wal
uuid: "70baca43719d4f988294386be9779ab3"
format_stamp: "Formatted at 2026-09-29 10:58:35 on dist-test-slave-8nrt"
server_key: "402fbac42776a7701f9ec3505fbc2134"
server_key_iv: "f25b45ba080a4355534a5d62df5fcf48"
server_key_version: "encryptionkey@0"
I20260929 10:58:35.929605 30934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:58:35.938306 30934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:58:35.938608 30934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:58:35.938766 30934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:58:35.939006 30934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:58:35.939330 30934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:58:35.939386 30934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:35.939426 30934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:58:35.939450 30934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:35.945214 30934 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.193:44219
I20260929 10:58:35.945328 31061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.193:44219 every 8 connection(s)
I20260929 10:58:35.945634 30934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/ts-0/data/info.pb
I20260929 10:58:35.950862 30914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:35.946385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.193:43183 (local address 127.10.238.254:35397)
0929 10:58:35.946430 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:35.946432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:35.946637 (+ 205us) server_negotiation.cc:407] Connection header received
0929 10:58:35.947183 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:35.947192 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:35.947209 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:35.947268 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:35.948026 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:35.948524 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:35.949550 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:35.949735 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:35.949788 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:35.950602 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:35.950603 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:35.950604 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:58:35.950608 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:35.950618 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:58:35.950621 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:35.950658 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:35.950777 (+ 119us) server_negotiation.cc:299] Negotiation successful
0929 10:58:35.950796 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 10:58:35.950934 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 30934
I20260929 10:58:35.951042 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/ts-0/wal/instance
I20260929 10:58:35.951223 31064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:35.946351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:35397 (local address 127.10.238.193:43183)
0929 10:58:35.946520 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:35.946553 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 10:58:35.947074 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:58:35.947336 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:58:35.947340 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:58:35.947350 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:58:35.947918 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:35.947925 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:35.948608 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:35.948609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:35.949476 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:35.949479 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:35.949813 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:35.949814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:35.949922 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:35.950478 (+ 556us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:58:35.950478 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:58:35.950506 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:58:35.950529 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:58:35.950673 (+ 144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:58:35.950678 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:58:35.950680 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 10:58:35.950777 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 10:58:35.950898 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":63,"threads_started":1}
I20260929 10:58:35.951272 11195 external_mini_cluster.cc:1428] Setting key 6a0590ee0d5c8d5a35b4e97a75960b1e
I20260929 10:58:35.951774 31062 heartbeater.cc:347] Connected to a master server at 127.10.238.254:35397
I20260929 10:58:35.951865 31062 heartbeater.cc:464] Registering TS with master...
I20260929 10:58:35.952045 31062 heartbeater.cc:511] Master 127.10.238.254:35397 requested a full tablet report, sending...
I20260929 10:58:35.952453 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.194:0
--local_ip_for_outbound_sockets=127.10.238.194
--webserver_interface=127.10.238.194
--webserver_port=0
--tserver_master_addrs=127.10.238.254:35397
--builtin_ntp_servers=127.10.238.212:39411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 10:58:35.952606 30873 ts_manager.cc:194] Registered new tserver with Master: 70baca43719d4f988294386be9779ab3 (127.10.238.193:44219)
I20260929 10:58:35.953181 30873 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.238.193:43183
W20260929 10:58:36.033761 31065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:58:36.033953 31065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:58:36.033975 31065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:58:36.035058 31065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:58:36.035091 31065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:58:36.035116 31065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.194
I20260929 10:58:36.036407 31065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:39411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.238.194
--webserver_port=0
--tserver_master_addrs=127.10.238.254:35397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.194
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:58:36.036590 31065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:58:36.036825 31065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:58:36.041955 31070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:36.041987 31071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:36.042152 31065 server_base.cc:1034] running on GCE node
W20260929 10:58:36.041987 31073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:36.042416 31065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:58:36.042668 31065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:58:36.043839 31065 hybrid_clock.cc:648] HybridClock initialized: now 1790679516043827 us; error 26 us; skew 500 ppm
I20260929 10:58:36.045147 31065 webserver.cc:460] Webserver started at http://127.10.238.194:39707/ using document root <none> and password file <none>
I20260929 10:58:36.045439 31065 fs_manager.cc:359] Metadata directory not provided
I20260929 10:58:36.045516 31065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:58:36.045666 31065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:58:36.046759 31065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/ts-1/data/instance:
uuid: "7f211e32380343fbad42d67d9b6543e9"
format_stamp: "Formatted at 2026-09-29 10:58:36 on dist-test-slave-8nrt"
server_key: "8290d1c49bc83a55225bef3a9d04f4bb"
server_key_iv: "8c4859b25a28ea2681aed1e40d67a50f"
server_key_version: "encryptionkey@0"
I20260929 10:58:36.047116 31065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/ts-1/wal/instance:
uuid: "7f211e32380343fbad42d67d9b6543e9"
format_stamp: "Formatted at 2026-09-29 10:58:36 on dist-test-slave-8nrt"
server_key: "8290d1c49bc83a55225bef3a9d04f4bb"
server_key_iv: "8c4859b25a28ea2681aed1e40d67a50f"
server_key_version: "encryptionkey@0"
I20260929 10:58:36.048394 31065 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 10:58:36.049196 31079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:36.049387 31065 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:58:36.049487 31065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/ts-1/wal
uuid: "7f211e32380343fbad42d67d9b6543e9"
format_stamp: "Formatted at 2026-09-29 10:58:36 on dist-test-slave-8nrt"
server_key: "8290d1c49bc83a55225bef3a9d04f4bb"
server_key_iv: "8c4859b25a28ea2681aed1e40d67a50f"
server_key_version: "encryptionkey@0"
I20260929 10:58:36.049569 31065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:58:36.071928 31065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:58:36.072346 31065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:58:36.072516 31065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:58:36.072768 31065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:58:36.073114 31065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:58:36.073171 31065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:36.073207 31065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:58:36.073233 31065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:36.079250 31065 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.194:35193
I20260929 10:58:36.079294 31192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.194:35193 every 8 connection(s)
I20260929 10:58:36.079638 31065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/ts-1/data/info.pb
I20260929 10:58:36.085203 30914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:36.080401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:38467 (local address 127.10.238.254:35397)
0929 10:58:36.080473 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:36.080475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:36.080761 (+ 286us) server_negotiation.cc:407] Connection header received
0929 10:58:36.081226 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:36.081227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:36.081240 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:36.081381 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:36.082094 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:36.082687 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:36.083798 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:36.084012 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:36.084098 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:36.084927 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:36.084929 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:36.084929 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:58:36.084934 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:36.084950 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:58:36.084954 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:36.085002 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:36.085094 (+ 92us) server_negotiation.cc:299] Negotiation successful
0929 10:58:36.085122 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 10:58:36.085489 31195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:36.080378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:35397 (local address 127.10.238.194:38467)
0929 10:58:36.080579 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:36.080620 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 10:58:36.081131 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:58:36.081460 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:58:36.081464 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:58:36.081472 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:58:36.081988 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:36.081992 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:36.082784 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:36.082785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:36.083684 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:36.083687 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:36.084113 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:36.084114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:36.084252 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:36.084784 (+ 532us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:58:36.084785 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:58:36.084814 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:58:36.084837 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:58:36.085002 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:58:36.085008 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:58:36.085011 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 10:58:36.085096 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 10:58:36.085201 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":85,"threads_started":1}
I20260929 10:58:36.085968 31193 heartbeater.cc:347] Connected to a master server at 127.10.238.254:35397
I20260929 10:58:36.086061 31193 heartbeater.cc:464] Registering TS with master...
I20260929 10:58:36.086211 31193 heartbeater.cc:511] Master 127.10.238.254:35397 requested a full tablet report, sending...
I20260929 10:58:36.086560 30873 ts_manager.cc:194] Registered new tserver with Master: 7f211e32380343fbad42d67d9b6543e9 (127.10.238.194:35193)
I20260929 10:58:36.086946 30873 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.238.194:38467
I20260929 10:58:36.087635 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 31065
I20260929 10:58:36.087754 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/ts-1/wal/instance
I20260929 10:58:36.087936 11195 external_mini_cluster.cc:1428] Setting key a8bafbeeb1e2107f0871c510b72ede91
I20260929 10:58:36.088845 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.238.195:0
--local_ip_for_outbound_sockets=127.10.238.195
--webserver_interface=127.10.238.195
--webserver_port=0
--tserver_master_addrs=127.10.238.254:35397
--builtin_ntp_servers=127.10.238.212:39411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 10:58:36.169952 31196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:58:36.170208 31196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:58:36.170249 31196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:58:36.171518 31196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:58:36.171583 31196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:58:36.171630 31196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.238.195
I20260929 10:58:36.173141 31196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:39411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.238.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.238.195
--webserver_port=0
--tserver_master_addrs=127.10.238.254:35397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.238.195
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:58:36.173450 31196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:58:36.173763 31196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:58:36.178957 31201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:36.178970 31202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:36.178960 31204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:36.179440 31196 server_base.cc:1034] running on GCE node
I20260929 10:58:36.179661 31196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:58:36.179919 31196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:58:36.181085 31196 hybrid_clock.cc:648] HybridClock initialized: now 1790679516181068 us; error 33 us; skew 500 ppm
I20260929 10:58:36.182670 31196 webserver.cc:460] Webserver started at http://127.10.238.195:45529/ using document root <none> and password file <none>
I20260929 10:58:36.182958 31196 fs_manager.cc:359] Metadata directory not provided
I20260929 10:58:36.183027 31196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:58:36.183167 31196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:58:36.184396 31196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/ts-2/data/instance:
uuid: "1f810b4f0bba43d0af3c2394760c4884"
format_stamp: "Formatted at 2026-09-29 10:58:36 on dist-test-slave-8nrt"
server_key: "a578a71f1ed0405e7a85f8e3be31ad5b"
server_key_iv: "7e056b0d8e944ac091563621949ca811"
server_key_version: "encryptionkey@0"
I20260929 10:58:36.184779 31196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/ts-2/wal/instance:
uuid: "1f810b4f0bba43d0af3c2394760c4884"
format_stamp: "Formatted at 2026-09-29 10:58:36 on dist-test-slave-8nrt"
server_key: "a578a71f1ed0405e7a85f8e3be31ad5b"
server_key_iv: "7e056b0d8e944ac091563621949ca811"
server_key_version: "encryptionkey@0"
I20260929 10:58:36.186162 31196 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 10:58:36.186892 31210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:36.187071 31196 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 10:58:36.187175 31196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/ts-2/wal
uuid: "1f810b4f0bba43d0af3c2394760c4884"
format_stamp: "Formatted at 2026-09-29 10:58:36 on dist-test-slave-8nrt"
server_key: "a578a71f1ed0405e7a85f8e3be31ad5b"
server_key_iv: "7e056b0d8e944ac091563621949ca811"
server_key_version: "encryptionkey@0"
I20260929 10:58:36.187252 31196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:58:36.210896 31196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:58:36.211264 31196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:58:36.211437 31196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:58:36.211695 31196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:58:36.212056 31196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:58:36.212122 31196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:36.212162 31196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:58:36.212208 31196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:36.218493 31196 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.195:34331
I20260929 10:58:36.218546 31323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.195:34331 every 8 connection(s)
I20260929 10:58:36.218948 31196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/ts-2/data/info.pb
I20260929 10:58:36.223615 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 31196
I20260929 10:58:36.223731 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/ts-2/wal/instance
I20260929 10:58:36.223964 11195 external_mini_cluster.cc:1428] Setting key 8f528d3534fa6a7450afd2c9941b8771
I20260929 10:58:36.224112 30914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:36.219688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.195:55253 (local address 127.10.238.254:35397)
0929 10:58:36.219783 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:36.219785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:36.220043 (+ 258us) server_negotiation.cc:407] Connection header received
0929 10:58:36.220485 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:36.220486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:36.220498 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:36.220562 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:36.221232 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:36.221818 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:36.222771 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:36.222950 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:36.222999 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:36.223801 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:36.223803 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:36.223803 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:58:36.223808 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:36.223828 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:58:36.223831 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:36.223876 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:36.223983 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 10:58:36.224021 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 10:58:36.224351 31326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:36.219640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:35397 (local address 127.10.238.195:55253)
0929 10:58:36.219850 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:36.219903 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 10:58:36.220395 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:58:36.220640 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:58:36.220644 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:58:36.220652 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:58:36.221149 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:36.221153 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:36.221896 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:36.221897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:36.222678 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:36.222679 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:36.223022 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:36.223023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:36.223130 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:36.223639 (+ 509us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:58:36.223640 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:58:36.223680 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:58:36.223713 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:58:36.223900 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:58:36.223906 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:58:36.223910 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 10:58:36.223982 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 10:58:36.224090 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":92,"threads_started":1}
I20260929 10:58:36.224766 31324 heartbeater.cc:347] Connected to a master server at 127.10.238.254:35397
I20260929 10:58:36.224879 31324 heartbeater.cc:464] Registering TS with master...
I20260929 10:58:36.225044 31324 heartbeater.cc:511] Master 127.10.238.254:35397 requested a full tablet report, sending...
I20260929 10:58:36.225399 30873 ts_manager.cc:194] Registered new tserver with Master: 1f810b4f0bba43d0af3c2394760c4884 (127.10.238.195:34331)
I20260929 10:58:36.225775 30873 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.238.195:55253
I20260929 10:58:36.235698 11195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:58:36.240103 30914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:36.237115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39836 (local address 127.10.238.254:35397)
0929 10:58:36.237190 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:36.237192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:36.237318 (+ 126us) server_negotiation.cc:407] Connection header received
0929 10:58:36.237421 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:36.237422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:36.237433 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:36.237504 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:36.237665 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:36.238167 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:36.238919 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:36.239120 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:36.239206 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:36.239833 (+ 627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:36.239835 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:36.239835 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:58:36.239840 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:36.239855 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:58:36.239859 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:36.239907 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:36.239995 (+ 88us) server_negotiation.cc:299] Negotiation successful
0929 10:58:36.240016 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 10:58:36.241745 30873 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39836:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:58:36.243301 30873 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:58:36.249449 31332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:36.245461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:44219 (local address 127.0.0.1:50356)
0929 10:58:36.245787 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:36.245831 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 10:58:36.245893 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:58:36.246194 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:58:36.246195 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:58:36.246197 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:58:36.246249 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:36.246250 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:36.247187 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:36.247188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:36.248296 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:36.248299 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:36.249130 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:36.249131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:36.249203 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:36.249215 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 10:58:36.249321 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 10:58:36.249374 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"spinlock_wait_cycles":768,"thread_start_us":106,"threads_started":1}
I20260929 10:58:36.249459 31334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:36.245550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50356 (local address 127.10.238.193:44219)
0929 10:58:36.245893 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:36.245897 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:36.245909 (+ 12us) server_negotiation.cc:407] Connection header received
0929 10:58:36.246029 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:36.246032 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:36.246050 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:36.246122 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:58:36.246334 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:36.247096 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:36.248390 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:36.249031 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:36.249128 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:36.249221 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:36.249337 (+ 116us) server_negotiation.cc:299] Negotiation successful
0929 10:58:36.249374 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":114,"threads_started":1}
I20260929 10:58:36.249766 31333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:36.245461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.195:34331 (local address 127.0.0.1:46394)
0929 10:58:36.245810 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:36.245844 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 10:58:36.245910 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:58:36.246131 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:58:36.246134 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:58:36.246136 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:58:36.246286 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:36.246288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:36.247605 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:36.247606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:36.248495 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:36.248497 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:36.249509 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:36.249510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:36.249586 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:36.249591 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 10:58:36.249653 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 10:58:36.249699 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":3,"thread_start_us":59,"threads_started":1}
I20260929 10:58:36.249912 31335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:36.245643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46394 (local address 127.10.238.195:34331)
0929 10:58:36.245808 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:36.245813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:36.245955 (+ 142us) server_negotiation.cc:407] Connection header received
0929 10:58:36.245997 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:36.245999 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:36.246015 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:36.246125 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:58:36.246369 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:36.247505 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:36.248581 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:36.249324 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:36.249392 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:36.249473 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:36.249690 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 10:58:36.249736 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":73,"threads_started":1}
I20260929 10:58:36.250211 30996 tablet_service.cc:1467] Processing CreateTablet for tablet a8bb95ea89d24588950789edc48b206c (DEFAULT_TABLE table=test-table [id=c986251ba6014e5891d748a30dfe9b8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:58:36.250579 30996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8bb95ea89d24588950789edc48b206c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:58:36.250618 31336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:36.245660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.194:35193 (local address 127.0.0.1:43684)
0929 10:58:36.246002 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:36.246014 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:58:36.246058 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:58:36.246385 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:58:36.246386 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:58:36.246388 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:58:36.246418 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:36.246420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:36.247647 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:36.247648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:36.248695 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:36.248697 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:36.250362 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:36.250363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:36.250427 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:36.250431 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 10:58:36.250498 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 10:58:36.250543 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":57,"threads_started":1}
I20260929 10:58:36.250643 31337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:36.245764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43684 (local address 127.10.238.194:35193)
0929 10:58:36.245949 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:36.245954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:36.246116 (+ 162us) server_negotiation.cc:407] Connection header received
0929 10:58:36.246223 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:36.246226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:36.246250 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:36.246320 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:58:36.246482 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:36.247564 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:36.248794 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:36.250270 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:36.250346 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:36.250426 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:36.250511 (+ 85us) server_negotiation.cc:299] Negotiation successful
0929 10:58:36.250553 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":75,"threads_started":1}
I20260929 10:58:36.250600 31258 tablet_service.cc:1467] Processing CreateTablet for tablet a8bb95ea89d24588950789edc48b206c (DEFAULT_TABLE table=test-table [id=c986251ba6014e5891d748a30dfe9b8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:58:36.250905 31258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8bb95ea89d24588950789edc48b206c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:58:36.253161 31338 tablet_bootstrap.cc:492] T a8bb95ea89d24588950789edc48b206c P 70baca43719d4f988294386be9779ab3: Bootstrap starting.
I20260929 10:58:36.253358 31339 tablet_bootstrap.cc:492] T a8bb95ea89d24588950789edc48b206c P 1f810b4f0bba43d0af3c2394760c4884: Bootstrap starting.
I20260929 10:58:36.253902 31338 tablet_bootstrap.cc:654] T a8bb95ea89d24588950789edc48b206c P 70baca43719d4f988294386be9779ab3: Neither blocks nor log segments found. Creating new log.
I20260929 10:58:36.253824 31127 tablet_service.cc:1467] Processing CreateTablet for tablet a8bb95ea89d24588950789edc48b206c (DEFAULT_TABLE table=test-table [id=c986251ba6014e5891d748a30dfe9b8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:58:36.254031 31339 tablet_bootstrap.cc:654] T a8bb95ea89d24588950789edc48b206c P 1f810b4f0bba43d0af3c2394760c4884: Neither blocks nor log segments found. Creating new log.
I20260929 10:58:36.254163 31127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8bb95ea89d24588950789edc48b206c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:58:36.254220 31338 log.cc:824] T a8bb95ea89d24588950789edc48b206c P 70baca43719d4f988294386be9779ab3: Log is configured to *not* fsync() on all Append() calls
I20260929 10:58:36.254345 31339 log.cc:824] T a8bb95ea89d24588950789edc48b206c P 1f810b4f0bba43d0af3c2394760c4884: Log is configured to *not* fsync() on all Append() calls
I20260929 10:58:36.255057 31339 tablet_bootstrap.cc:492] T a8bb95ea89d24588950789edc48b206c P 1f810b4f0bba43d0af3c2394760c4884: No bootstrap required, opened a new log
I20260929 10:58:36.255147 31339 ts_tablet_manager.cc:1397] T a8bb95ea89d24588950789edc48b206c P 1f810b4f0bba43d0af3c2394760c4884: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 10:58:36.257241 31342 tablet_bootstrap.cc:492] T a8bb95ea89d24588950789edc48b206c P 7f211e32380343fbad42d67d9b6543e9: Bootstrap starting.
I20260929 10:58:36.257658 31339 raft_consensus.cc:348] T a8bb95ea89d24588950789edc48b206c P 1f810b4f0bba43d0af3c2394760c4884 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70baca43719d4f988294386be9779ab3" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 44219 } } peers { permanent_uuid: "1f810b4f0bba43d0af3c2394760c4884" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 34331 } } peers { permanent_uuid: "7f211e32380343fbad42d67d9b6543e9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35193 } }
I20260929 10:58:36.257812 31339 raft_consensus.cc:374] T a8bb95ea89d24588950789edc48b206c P 1f810b4f0bba43d0af3c2394760c4884 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:58:36.257856 31339 raft_consensus.cc:729] T a8bb95ea89d24588950789edc48b206c P 1f810b4f0bba43d0af3c2394760c4884 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f810b4f0bba43d0af3c2394760c4884, State: Initialized, Role: FOLLOWER
I20260929 10:58:36.257912 31338 tablet_bootstrap.cc:492] T a8bb95ea89d24588950789edc48b206c P 70baca43719d4f988294386be9779ab3: No bootstrap required, opened a new log
I20260929 10:58:36.257974 31338 ts_tablet_manager.cc:1397] T a8bb95ea89d24588950789edc48b206c P 70baca43719d4f988294386be9779ab3: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260929 10:58:36.257975 31339 consensus_queue.cc:260] T a8bb95ea89d24588950789edc48b206c P 1f810b4f0bba43d0af3c2394760c4884 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70baca43719d4f988294386be9779ab3" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 44219 } } peers { permanent_uuid: "1f810b4f0bba43d0af3c2394760c4884" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 34331 } } peers { permanent_uuid: "7f211e32380343fbad42d67d9b6543e9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35193 } }
I20260929 10:58:36.258280 31339 ts_tablet_manager.cc:1428] T a8bb95ea89d24588950789edc48b206c P 1f810b4f0bba43d0af3c2394760c4884: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 10:58:36.258328 31324 heartbeater.cc:503] Master 127.10.238.254:35397 was elected leader, sending a full tablet report...
I20260929 10:58:36.260355 31338 raft_consensus.cc:348] T a8bb95ea89d24588950789edc48b206c P 70baca43719d4f988294386be9779ab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70baca43719d4f988294386be9779ab3" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 44219 } } peers { permanent_uuid: "1f810b4f0bba43d0af3c2394760c4884" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 34331 } } peers { permanent_uuid: "7f211e32380343fbad42d67d9b6543e9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35193 } }
I20260929 10:58:36.260483 31338 raft_consensus.cc:374] T a8bb95ea89d24588950789edc48b206c P 70baca43719d4f988294386be9779ab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:58:36.260504 31338 raft_consensus.cc:729] T a8bb95ea89d24588950789edc48b206c P 70baca43719d4f988294386be9779ab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70baca43719d4f988294386be9779ab3, State: Initialized, Role: FOLLOWER
I20260929 10:58:36.260615 31338 consensus_queue.cc:260] T a8bb95ea89d24588950789edc48b206c P 70baca43719d4f988294386be9779ab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70baca43719d4f988294386be9779ab3" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 44219 } } peers { permanent_uuid: "1f810b4f0bba43d0af3c2394760c4884" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 34331 } } peers { permanent_uuid: "7f211e32380343fbad42d67d9b6543e9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35193 } }
I20260929 10:58:36.260875 31338 ts_tablet_manager.cc:1428] T a8bb95ea89d24588950789edc48b206c P 70baca43719d4f988294386be9779ab3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 10:58:36.260938 31062 heartbeater.cc:503] Master 127.10.238.254:35397 was elected leader, sending a full tablet report...
I20260929 10:58:36.261359 31342 tablet_bootstrap.cc:654] T a8bb95ea89d24588950789edc48b206c P 7f211e32380343fbad42d67d9b6543e9: Neither blocks nor log segments found. Creating new log.
I20260929 10:58:36.261730 31342 log.cc:824] T a8bb95ea89d24588950789edc48b206c P 7f211e32380343fbad42d67d9b6543e9: Log is configured to *not* fsync() on all Append() calls
I20260929 10:58:36.262543 31342 tablet_bootstrap.cc:492] T a8bb95ea89d24588950789edc48b206c P 7f211e32380343fbad42d67d9b6543e9: No bootstrap required, opened a new log
I20260929 10:58:36.262598 31342 ts_tablet_manager.cc:1397] T a8bb95ea89d24588950789edc48b206c P 7f211e32380343fbad42d67d9b6543e9: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20260929 10:58:36.264511 31342 raft_consensus.cc:348] T a8bb95ea89d24588950789edc48b206c P 7f211e32380343fbad42d67d9b6543e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70baca43719d4f988294386be9779ab3" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 44219 } } peers { permanent_uuid: "1f810b4f0bba43d0af3c2394760c4884" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 34331 } } peers { permanent_uuid: "7f211e32380343fbad42d67d9b6543e9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35193 } }
I20260929 10:58:36.264639 31342 raft_consensus.cc:374] T a8bb95ea89d24588950789edc48b206c P 7f211e32380343fbad42d67d9b6543e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:58:36.264660 31342 raft_consensus.cc:729] T a8bb95ea89d24588950789edc48b206c P 7f211e32380343fbad42d67d9b6543e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f211e32380343fbad42d67d9b6543e9, State: Initialized, Role: FOLLOWER
I20260929 10:58:36.264747 31342 consensus_queue.cc:260] T a8bb95ea89d24588950789edc48b206c P 7f211e32380343fbad42d67d9b6543e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70baca43719d4f988294386be9779ab3" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 44219 } } peers { permanent_uuid: "1f810b4f0bba43d0af3c2394760c4884" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 34331 } } peers { permanent_uuid: "7f211e32380343fbad42d67d9b6543e9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35193 } }
I20260929 10:58:36.264983 31342 ts_tablet_manager.cc:1428] T a8bb95ea89d24588950789edc48b206c P 7f211e32380343fbad42d67d9b6543e9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 10:58:36.265043 31193 heartbeater.cc:503] Master 127.10.238.254:35397 was elected leader, sending a full tablet report...
I20260929 10:58:36.267201 31346 raft_consensus.cc:482] T a8bb95ea89d24588950789edc48b206c P 7f211e32380343fbad42d67d9b6543e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:58:36.267282 31346 raft_consensus.cc:504] T a8bb95ea89d24588950789edc48b206c P 7f211e32380343fbad42d67d9b6543e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70baca43719d4f988294386be9779ab3" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 44219 } } peers { permanent_uuid: "1f810b4f0bba43d0af3c2394760c4884" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 34331 } } peers { permanent_uuid: "7f211e32380343fbad42d67d9b6543e9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35193 } }
I20260929 10:58:36.267544 31346 leader_election.cc:288] T a8bb95ea89d24588950789edc48b206c P 7f211e32380343fbad42d67d9b6543e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70baca43719d4f988294386be9779ab3 (127.10.238.193:44219), 1f810b4f0bba43d0af3c2394760c4884 (127.10.238.195:34331)
I20260929 10:58:36.270845 31334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:36.267715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:56833 (local address 127.10.238.193:44219)
0929 10:58:36.267765 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:36.267766 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:36.267819 (+ 53us) server_negotiation.cc:407] Connection header received
0929 10:58:36.267904 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:36.267905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:36.267917 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:36.267969 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:58:36.268165 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:36.268925 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:36.270088 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:36.270470 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:36.270560 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:36.270569 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:36.270741 (+ 172us) server_negotiation.cc:299] Negotiation successful
0929 10:58:36.270781 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 10:58:36.270845 31195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:36.267672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:44219 (local address 127.10.238.194:56833)
0929 10:58:36.267768 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:36.267780 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:58:36.267836 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:58:36.267988 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:58:36.267989 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:58:36.267991 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:58:36.268088 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:36.268089 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:36.269006 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:36.269007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:36.269978 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:36.269981 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:36.270559 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:36.270560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:36.270630 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:36.270634 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 10:58:36.270733 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 10:58:36.270777 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"mutex_wait_us":24}
I20260929 10:58:36.270926 31335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:36.267893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.238.194:42873 (local address 127.10.238.195:34331)
0929 10:58:36.267942 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:36.267944 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:36.267952 (+ 8us) server_negotiation.cc:407] Connection header received
0929 10:58:36.267990 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:36.267991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:36.268010 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:36.268064 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:58:36.268225 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:36.268922 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:36.270188 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:36.270553 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:36.270648 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:36.270655 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:36.270821 (+ 166us) server_negotiation.cc:299] Negotiation successful
0929 10:58:36.270860 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 10:58:36.270977 31347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:36.267677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.195:34331 (local address 127.10.238.194:42873)
0929 10:58:36.267848 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:36.267859 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 10:58:36.267911 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:58:36.268092 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:58:36.268092 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:58:36.268094 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:58:36.268155 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:36.268156 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:36.269007 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:36.269008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:36.270094 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:58:36.270098 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:58:36.270651 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:58:36.270652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:58:36.270733 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:36.270735 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 10:58:36.270804 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 10:58:36.270842 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":46,"thread_start_us":103,"threads_started":1}
I20260929 10:58:36.271262 31016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8bb95ea89d24588950789edc48b206c" candidate_uuid: "7f211e32380343fbad42d67d9b6543e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70baca43719d4f988294386be9779ab3" is_pre_election: true
I20260929 10:58:36.271297 31278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8bb95ea89d24588950789edc48b206c" candidate_uuid: "7f211e32380343fbad42d67d9b6543e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f810b4f0bba43d0af3c2394760c4884" is_pre_election: true
I20260929 10:58:36.271399 31278 raft_consensus.cc:2454] T a8bb95ea89d24588950789edc48b206c P 1f810b4f0bba43d0af3c2394760c4884 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f211e32380343fbad42d67d9b6543e9 in term 0.
I20260929 10:58:36.271394 31016 raft_consensus.cc:2454] T a8bb95ea89d24588950789edc48b206c P 70baca43719d4f988294386be9779ab3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f211e32380343fbad42d67d9b6543e9 in term 0.
I20260929 10:58:36.271592 31080 leader_election.cc:302] T a8bb95ea89d24588950789edc48b206c P 7f211e32380343fbad42d67d9b6543e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f810b4f0bba43d0af3c2394760c4884, 7f211e32380343fbad42d67d9b6543e9; no voters:
I20260929 10:58:36.271710 31346 raft_consensus.cc:2781] T a8bb95ea89d24588950789edc48b206c P 7f211e32380343fbad42d67d9b6543e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:58:36.271770 31346 raft_consensus.cc:482] T a8bb95ea89d24588950789edc48b206c P 7f211e32380343fbad42d67d9b6543e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:58:36.271795 31346 raft_consensus.cc:3037] T a8bb95ea89d24588950789edc48b206c P 7f211e32380343fbad42d67d9b6543e9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:58:36.272449 31346 raft_consensus.cc:504] T a8bb95ea89d24588950789edc48b206c P 7f211e32380343fbad42d67d9b6543e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70baca43719d4f988294386be9779ab3" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 44219 } } peers { permanent_uuid: "1f810b4f0bba43d0af3c2394760c4884" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 34331 } } peers { permanent_uuid: "7f211e32380343fbad42d67d9b6543e9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35193 } }
I20260929 10:58:36.272605 31346 leader_election.cc:288] T a8bb95ea89d24588950789edc48b206c P 7f211e32380343fbad42d67d9b6543e9 [CANDIDATE]: Term 1 election: Requested vote from peers 70baca43719d4f988294386be9779ab3 (127.10.238.193:44219), 1f810b4f0bba43d0af3c2394760c4884 (127.10.238.195:34331)
I20260929 10:58:36.272750 31016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8bb95ea89d24588950789edc48b206c" candidate_uuid: "7f211e32380343fbad42d67d9b6543e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70baca43719d4f988294386be9779ab3"
I20260929 10:58:36.272789 31278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8bb95ea89d24588950789edc48b206c" candidate_uuid: "7f211e32380343fbad42d67d9b6543e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f810b4f0bba43d0af3c2394760c4884"
I20260929 10:58:36.272832 31016 raft_consensus.cc:3037] T a8bb95ea89d24588950789edc48b206c P 70baca43719d4f988294386be9779ab3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:58:36.272882 31278 raft_consensus.cc:3037] T a8bb95ea89d24588950789edc48b206c P 1f810b4f0bba43d0af3c2394760c4884 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:58:36.273636 31016 raft_consensus.cc:2454] T a8bb95ea89d24588950789edc48b206c P 70baca43719d4f988294386be9779ab3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f211e32380343fbad42d67d9b6543e9 in term 1.
I20260929 10:58:36.273651 31278 raft_consensus.cc:2454] T a8bb95ea89d24588950789edc48b206c P 1f810b4f0bba43d0af3c2394760c4884 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f211e32380343fbad42d67d9b6543e9 in term 1.
I20260929 10:58:36.273810 31080 leader_election.cc:302] T a8bb95ea89d24588950789edc48b206c P 7f211e32380343fbad42d67d9b6543e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f810b4f0bba43d0af3c2394760c4884, 7f211e32380343fbad42d67d9b6543e9; no voters:
I20260929 10:58:36.273911 31346 raft_consensus.cc:2781] T a8bb95ea89d24588950789edc48b206c P 7f211e32380343fbad42d67d9b6543e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:58:36.274045 31346 raft_consensus.cc:686] T a8bb95ea89d24588950789edc48b206c P 7f211e32380343fbad42d67d9b6543e9 [term 1 LEADER]: Becoming Leader. State: Replica: 7f211e32380343fbad42d67d9b6543e9, State: Running, Role: LEADER
I20260929 10:58:36.274137 31346 consensus_queue.cc:237] T a8bb95ea89d24588950789edc48b206c P 7f211e32380343fbad42d67d9b6543e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70baca43719d4f988294386be9779ab3" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 44219 } } peers { permanent_uuid: "1f810b4f0bba43d0af3c2394760c4884" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 34331 } } peers { permanent_uuid: "7f211e32380343fbad42d67d9b6543e9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35193 } }
I20260929 10:58:36.274911 30873 catalog_manager.cc:5495] T a8bb95ea89d24588950789edc48b206c P 7f211e32380343fbad42d67d9b6543e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f211e32380343fbad42d67d9b6543e9 (127.10.238.194). New cstate: current_term: 1 leader_uuid: "7f211e32380343fbad42d67d9b6543e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70baca43719d4f988294386be9779ab3" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 44219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f810b4f0bba43d0af3c2394760c4884" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 34331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f211e32380343fbad42d67d9b6543e9" member_type: VOTER last_known_addr { host: "127.10.238.194" port: 35193 } health_report { overall_health: HEALTHY } } }
I20260929 10:58:36.279620 31337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:36.277227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43700 (local address 127.10.238.194:35193)
0929 10:58:36.277297 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:36.277299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:36.277308 (+ 9us) server_negotiation.cc:407] Connection header received
0929 10:58:36.277384 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:36.277384 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:36.277403 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:36.277469 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:58:36.277606 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:36.278055 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:36.278821 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:36.279028 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:36.279132 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:36.279301 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:58:36.279397 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:58:36.279484 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:36.279511 (+ 27us) server_negotiation.cc:299] Negotiation successful
0929 10:58:36.279533 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 10:58:36.281234 31016 raft_consensus.cc:1261] T a8bb95ea89d24588950789edc48b206c P 70baca43719d4f988294386be9779ab3 [term 1 FOLLOWER]: Refusing update from remote peer 7f211e32380343fbad42d67d9b6543e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:58:36.281602 31278 raft_consensus.cc:1261] T a8bb95ea89d24588950789edc48b206c P 1f810b4f0bba43d0af3c2394760c4884 [term 1 FOLLOWER]: Refusing update from remote peer 7f211e32380343fbad42d67d9b6543e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:58:36.281605 31348 consensus_queue.cc:1035] T a8bb95ea89d24588950789edc48b206c P 7f211e32380343fbad42d67d9b6543e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70baca43719d4f988294386be9779ab3" member_type: VOTER last_known_addr { host: "127.10.238.193" port: 44219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:58:36.281888 31348 consensus_queue.cc:1035] T a8bb95ea89d24588950789edc48b206c P 7f211e32380343fbad42d67d9b6543e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f810b4f0bba43d0af3c2394760c4884" member_type: VOTER last_known_addr { host: "127.10.238.195" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:58:36.284446 31353 mvcc.cc:205] Tried to move back new op lower bound from 7334623298685427712 to 7334623298659565568. Current Snapshot: MvccSnapshot[applied={T|T < 7334623298685427712}]
I20260929 10:58:36.284696 31352 mvcc.cc:205] Tried to move back new op lower bound from 7334623298685427712 to 7334623298659565568. Current Snapshot: MvccSnapshot[applied={T|T < 7334623298685427712}]
I20260929 10:58:36.286602 30873 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:58:36.286698 30873 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:58:36.287415 30873 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f19af86671f2492f81f5f729d739b24a: Sending DeleteTablet for 3 replicas of tablet a8bb95ea89d24588950789edc48b206c
I20260929 10:58:36.287709 30996 tablet_service.cc:1514] Processing DeleteTablet for tablet a8bb95ea89d24588950789edc48b206c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:58:36 UTC) from {username='slave'} at 127.0.0.1:50356
I20260929 10:58:36.287789 31258 tablet_service.cc:1514] Processing DeleteTablet for tablet a8bb95ea89d24588950789edc48b206c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:58:36 UTC) from {username='slave'} at 127.0.0.1:46394
I20260929 10:58:36.287941 31361 tablet_replica.cc:330] T a8bb95ea89d24588950789edc48b206c P 70baca43719d4f988294386be9779ab3: stopping tablet replica
I20260929 10:58:36.288035 31361 raft_consensus.cc:2229] T a8bb95ea89d24588950789edc48b206c P 70baca43719d4f988294386be9779ab3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:58:36.288067 31127 tablet_service.cc:1514] Processing DeleteTablet for tablet a8bb95ea89d24588950789edc48b206c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:58:36 UTC) from {username='slave'} at 127.0.0.1:43684
I20260929 10:58:36.288125 31361 raft_consensus.cc:2258] T a8bb95ea89d24588950789edc48b206c P 70baca43719d4f988294386be9779ab3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:58:36.288400 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 30934
I20260929 10:58:36.288553 31361 ts_tablet_manager.cc:1905] T a8bb95ea89d24588950789edc48b206c P 70baca43719d4f988294386be9779ab3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:58:36.289018 31362 tablet_replica.cc:330] T a8bb95ea89d24588950789edc48b206c P 1f810b4f0bba43d0af3c2394760c4884: stopping tablet replica
I20260929 10:58:36.289127 31362 raft_consensus.cc:2229] T a8bb95ea89d24588950789edc48b206c P 1f810b4f0bba43d0af3c2394760c4884 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:58:36.289229 31362 raft_consensus.cc:2258] T a8bb95ea89d24588950789edc48b206c P 1f810b4f0bba43d0af3c2394760c4884 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:58:36.289697 31362 ts_tablet_manager.cc:1905] T a8bb95ea89d24588950789edc48b206c P 1f810b4f0bba43d0af3c2394760c4884: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:58:36.291285 31362 ts_tablet_manager.cc:1918] T a8bb95ea89d24588950789edc48b206c P 1f810b4f0bba43d0af3c2394760c4884: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:58:36.291378 31362 log.cc:1196] T a8bb95ea89d24588950789edc48b206c P 1f810b4f0bba43d0af3c2394760c4884: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/ts-2/wal/wals/a8bb95ea89d24588950789edc48b206c
I20260929 10:58:36.291658 31362 ts_tablet_manager.cc:1939] T a8bb95ea89d24588950789edc48b206c P 1f810b4f0bba43d0af3c2394760c4884: Deleting consensus metadata
I20260929 10:58:36.292084 30858 catalog_manager.cc:4841] TS 1f810b4f0bba43d0af3c2394760c4884 (127.10.238.195:34331): tablet a8bb95ea89d24588950789edc48b206c (table test-table [id=c986251ba6014e5891d748a30dfe9b8e]) successfully deleted
I20260929 10:58:36.293515 31363 tablet_replica.cc:330] T a8bb95ea89d24588950789edc48b206c P 7f211e32380343fbad42d67d9b6543e9: stopping tablet replica
I20260929 10:58:36.293637 31363 raft_consensus.cc:2229] T a8bb95ea89d24588950789edc48b206c P 7f211e32380343fbad42d67d9b6543e9 [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:58:36.293777 31363 raft_consensus.cc:2258] T a8bb95ea89d24588950789edc48b206c P 7f211e32380343fbad42d67d9b6543e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:58:36.294157 31363 ts_tablet_manager.cc:1905] T a8bb95ea89d24588950789edc48b206c P 7f211e32380343fbad42d67d9b6543e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:58:36.295663 31363 ts_tablet_manager.cc:1918] T a8bb95ea89d24588950789edc48b206c P 7f211e32380343fbad42d67d9b6543e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:58:36.295765 31363 log.cc:1196] T a8bb95ea89d24588950789edc48b206c P 7f211e32380343fbad42d67d9b6543e9: Deleting WAL directory at /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790679274510869-11195-0/minicluster-data/ts-1/wal/wals/a8bb95ea89d24588950789edc48b206c
I20260929 10:58:36.296010 31363 ts_tablet_manager.cc:1939] T a8bb95ea89d24588950789edc48b206c P 7f211e32380343fbad42d67d9b6543e9: Deleting consensus metadata
I20260929 10:58:36.296432 30858 catalog_manager.cc:4841] TS 7f211e32380343fbad42d67d9b6543e9 (127.10.238.194:35193): tablet a8bb95ea89d24588950789edc48b206c (table test-table [id=c986251ba6014e5891d748a30dfe9b8e]) successfully deleted
W20260929 10:58:36.296591 30859 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.238.193:44219 (error 108)
I20260929 10:58:36.296814 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 31065
I20260929 10:58:36.297509 31336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:36.296923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.193:44219 (local address 127.0.0.1:50360)
0929 10:58:36.297437 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0929 10:58:36.297480 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.193:44219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494}
W20260929 10:58:36.297685 30859 catalog_manager.cc:4584] TS 70baca43719d4f988294386be9779ab3 (127.10.238.193:44219): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a8bb95ea89d24588950789edc48b206c: Network error: Client connection negotiation failed: client connection to 127.10.238.193:44219: connect: Connection refused (error 111)
I20260929 10:58:36.302587 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 31196
I20260929 10:58:36.307308 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 30843
2026-09-29T10:58:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (671 ms)
[----------] 6 tests from EncryptionPolicyTest (8688 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790679274510869-11195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:58:36 dist-test-slave-8nrt krb5kdc[31367](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:58:36 dist-test-slave-8nrt krb5kdc[31367](info): set up 2 sockets
Sep 29 10:58:36 dist-test-slave-8nrt krb5kdc[31367](info): commencing operation
krb5kdc: starting...
W20260929 10:58:37.377522 11195 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Sep 29 10:58:37 dist-test-slave-8nrt krb5kdc[31367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:58:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:58:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790679274510869-11195-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790679274510869-11195-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 10:58:37.496381 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36869
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790679274510869-11195-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36869
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:58:37.579417 31380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:58:37.579664 31380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:58:37.579691 31380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:58:37.580875 31380 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:58:37.580925 31380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:58:37.580941 31380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:58:37.580984 31380 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:58:37.581020 31380 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:58:37.582540 31380 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790679274510869-11195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36869
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790679274510869-11195-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:58:37.582815 31380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:58:37.583073 31380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:58:37.588243 31386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:37.588271 31388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:37.588271 31385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:37.588294 31380 server_base.cc:1034] running on GCE node
I20260929 10:58:37.588758 31380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:58:37.589013 31380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:58:37.590209 31380 hybrid_clock.cc:648] HybridClock initialized: now 1790679517590171 us; error 57 us; skew 500 ppm
Sep 29 10:58:37 dist-test-slave-8nrt krb5kdc[31367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:58:37.592792 31380 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 10:58:37.593233 31380 webserver.cc:460] Webserver started at http://127.0.0.1:43907/ using document root <none> and password file <none>
I20260929 10:58:37.593538 31380 fs_manager.cc:359] Metadata directory not provided
I20260929 10:58:37.593586 31380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:58:37.593737 31380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:58:37.594823 31380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790679274510869-11195-0/minicluster-data/master-0/data/instance:
uuid: "ccdd3c324f664014922f5f883346a61a"
format_stamp: "Formatted at 2026-09-29 10:58:37 on dist-test-slave-8nrt"
server_key: "bfc69554f365144c79ceb24ac3532a59"
server_key_iv: "4796556efbd721ac90c551eb7ca0a3ce"
server_key_version: "encryptionkey@0"
I20260929 10:58:37.595196 31380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790679274510869-11195-0/minicluster-data/master-0/wal/instance:
uuid: "ccdd3c324f664014922f5f883346a61a"
format_stamp: "Formatted at 2026-09-29 10:58:37 on dist-test-slave-8nrt"
server_key: "bfc69554f365144c79ceb24ac3532a59"
server_key_iv: "4796556efbd721ac90c551eb7ca0a3ce"
server_key_version: "encryptionkey@0"
I20260929 10:58:37.596529 31380 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 10:58:37.597335 31395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:37.597513 31380 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 10:58:37.597611 31380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "ccdd3c324f664014922f5f883346a61a"
format_stamp: "Formatted at 2026-09-29 10:58:37 on dist-test-slave-8nrt"
server_key: "bfc69554f365144c79ceb24ac3532a59"
server_key_iv: "4796556efbd721ac90c551eb7ca0a3ce"
server_key_version: "encryptionkey@0"
I20260929 10:58:37.597695 31380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:58:37.631723 31380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:58:37.633841 31380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:58:37.634008 31380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:58:37.638425 31380 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36869
I20260929 10:58:37.638477 31447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36869 every 8 connection(s)
I20260929 10:58:37.638860 31380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:58:37.639791 31448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:58:37.641458 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 31380
I20260929 10:58:37.641588 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790679274510869-11195-0/minicluster-data/master-0/wal/instance
I20260929 10:58:37.641836 11195 external_mini_cluster.cc:1428] Setting key 95ecbf7ed94f3e6653e49860e9790073
I20260929 10:58:37.642409 31448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccdd3c324f664014922f5f883346a61a: Bootstrap starting.
I20260929 10:58:37.643110 31448 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccdd3c324f664014922f5f883346a61a: Neither blocks nor log segments found. Creating new log.
I20260929 10:58:37.643443 31448 log.cc:824] T 00000000000000000000000000000000 P ccdd3c324f664014922f5f883346a61a: Log is configured to *not* fsync() on all Append() calls
I20260929 10:58:37.644208 31448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccdd3c324f664014922f5f883346a61a: No bootstrap required, opened a new log
I20260929 10:58:37.646775 31448 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccdd3c324f664014922f5f883346a61a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccdd3c324f664014922f5f883346a61a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36869 } }
I20260929 10:58:37.646945 31448 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccdd3c324f664014922f5f883346a61a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:58:37.646977 31448 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccdd3c324f664014922f5f883346a61a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccdd3c324f664014922f5f883346a61a, State: Initialized, Role: FOLLOWER
I20260929 10:58:37.647094 31448 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccdd3c324f664014922f5f883346a61a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccdd3c324f664014922f5f883346a61a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36869 } }
I20260929 10:58:37.647176 31448 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccdd3c324f664014922f5f883346a61a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:58:37.647205 31448 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccdd3c324f664014922f5f883346a61a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:58:37.647249 31448 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccdd3c324f664014922f5f883346a61a [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:58:37.648029 31448 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccdd3c324f664014922f5f883346a61a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccdd3c324f664014922f5f883346a61a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36869 } }
I20260929 10:58:37.648128 31448 leader_election.cc:302] T 00000000000000000000000000000000 P ccdd3c324f664014922f5f883346a61a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccdd3c324f664014922f5f883346a61a; no voters:
I20260929 10:58:37.648341 31448 leader_election.cc:288] T 00000000000000000000000000000000 P ccdd3c324f664014922f5f883346a61a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:58:37.648367 31453 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccdd3c324f664014922f5f883346a61a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:58:37.648480 31453 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccdd3c324f664014922f5f883346a61a [term 1 LEADER]: Becoming Leader. State: Replica: ccdd3c324f664014922f5f883346a61a, State: Running, Role: LEADER
I20260929 10:58:37.648619 31448 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccdd3c324f664014922f5f883346a61a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:58:37.648643 31453 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccdd3c324f664014922f5f883346a61a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccdd3c324f664014922f5f883346a61a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36869 } }
I20260929 10:58:37.648977 31454 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccdd3c324f664014922f5f883346a61a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ccdd3c324f664014922f5f883346a61a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccdd3c324f664014922f5f883346a61a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36869 } } }
I20260929 10:58:37.649178 31454 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccdd3c324f664014922f5f883346a61a [sys.catalog]: This master's current role is: LEADER
I20260929 10:58:37.648965 31453 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccdd3c324f664014922f5f883346a61a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccdd3c324f664014922f5f883346a61a. Latest consensus state: current_term: 1 leader_uuid: "ccdd3c324f664014922f5f883346a61a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccdd3c324f664014922f5f883346a61a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36869 } } }
I20260929 10:58:37.649720 31453 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccdd3c324f664014922f5f883346a61a [sys.catalog]: This master's current role is: LEADER
Sep 29 10:58:37 dist-test-slave-8nrt krb5kdc[31367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 10:58:37.649948 31462 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:58:37.650528 31462 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:58:37.652685 31462 catalog_manager.cc:1347] Generated new cluster ID: e37d50a7fd534949aac347125981e4c5
I20260929 10:58:37.652742 31462 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:58:37.654477 31451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:37.642467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57597 (local address 127.0.0.1:36869)
0929 10:58:37.642676 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:37.642693 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:37.642732 (+ 39us) server_negotiation.cc:407] Connection header received
0929 10:58:37.643350 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:37.643360 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:37.643431 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:37.643587 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:37.644355 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:37.645886 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:37.646691 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:37.647291 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:37.647405 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:37.650493 (+ 3088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:37.650497 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:37.650498 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:58:37.650513 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:37.652699 (+ 2186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:37.653208 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:37.653211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:37.653211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:37.653343 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:37.653598 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:37.653599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:37.653600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:37.653867 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:37.654063 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:37.654150 (+ 87us) server_negotiation.cc:299] Negotiation successful
0929 10:58:37.654258 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":93,"threads_started":1}
I20260929 10:58:37.655511 11195 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:58:37.667897 31451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:37.657550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48443 (local address 127.0.0.1:36869)
0929 10:58:37.657588 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:37.657590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:37.661460 (+ 3870us) server_negotiation.cc:407] Connection header received
0929 10:58:37.661535 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:37.661536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:37.661545 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:37.661597 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:37.662164 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:37.662682 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:37.663581 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:37.663841 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:37.663934 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:37.665160 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:37.665161 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:37.665161 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:58:37.665168 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:37.666647 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:37.667025 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:37.667026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:37.667026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:37.667088 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:37.667311 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:37.667312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:37.667313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:37.667426 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:37.667538 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:37.667725 (+ 187us) server_negotiation.cc:299] Negotiation successful
0929 10:58:37.667790 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 10:58:37.671095 31462 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:58:37.671588 31462 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:58:37.675109 31462 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccdd3c324f664014922f5f883346a61a: Generated new TSK 0
I20260929 10:58:37.675302 31462 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:58:37.679008 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 31380
W20260929 10:58:37.683329 31365 connection.cc:535] client connection to 127.0.0.1:36869 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36869): Connection reset by peer (error 104)
2026-09-29T10:58:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1369 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790679274510869-11195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:58:37 dist-test-slave-8nrt krb5kdc[31479](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:58:37 dist-test-slave-8nrt krb5kdc[31479](info): set up 2 sockets
Sep 29 10:58:37 dist-test-slave-8nrt krb5kdc[31479](info): commencing operation
krb5kdc: starting...
W20260929 10:58:38.721428 11195 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Sep 29 10:58:38 dist-test-slave-8nrt krb5kdc[31479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:58:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:58:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790679274510869-11195-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790679274510869-11195-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 10:58:38.844851 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36689
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790679274510869-11195-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:58:38.924904 31492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:58:38.925148 31492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:58:38.925185 31492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:58:38.926349 31492 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:58:38.926429 31492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:58:38.926453 31492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:58:38.926474 31492 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:58:38.926501 31492 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:58:38.927983 31492 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790679274510869-11195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36689
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790679274510869-11195-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:58:38.928272 31492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:58:38.928501 31492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:58:38.933549 31497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:38.933562 31500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:38.933615 31498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:38.933699 31492 server_base.cc:1034] running on GCE node
I20260929 10:58:38.934074 31492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:58:38.934338 31492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:58:38.935490 31492 hybrid_clock.cc:648] HybridClock initialized: now 1790679518935467 us; error 37 us; skew 500 ppm
Sep 29 10:58:38 dist-test-slave-8nrt krb5kdc[31479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679518, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:58:38.937999 31492 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 10:58:38.938482 31492 webserver.cc:460] Webserver started at http://127.0.0.1:39283/ using document root <none> and password file <none>
I20260929 10:58:38.938720 31492 fs_manager.cc:359] Metadata directory not provided
I20260929 10:58:38.938767 31492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:58:38.938935 31492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:58:38.940008 31492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790679274510869-11195-0/minicluster-data/master-0/data/instance:
uuid: "1ff01e7d4f924fd5b0b17d0b7eb4349b"
format_stamp: "Formatted at 2026-09-29 10:58:38 on dist-test-slave-8nrt"
server_key: "1f306cf4d7ead1e241a4fd3190dc0888"
server_key_iv: "aa43ec364b264faa99b49ec351bf6154"
server_key_version: "encryptionkey@0"
I20260929 10:58:38.940377 31492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790679274510869-11195-0/minicluster-data/master-0/wal/instance:
uuid: "1ff01e7d4f924fd5b0b17d0b7eb4349b"
format_stamp: "Formatted at 2026-09-29 10:58:38 on dist-test-slave-8nrt"
server_key: "1f306cf4d7ead1e241a4fd3190dc0888"
server_key_iv: "aa43ec364b264faa99b49ec351bf6154"
server_key_version: "encryptionkey@0"
I20260929 10:58:38.941814 31492 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 10:58:38.942576 31507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:38.942790 31492 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 10:58:38.942873 31492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "1ff01e7d4f924fd5b0b17d0b7eb4349b"
format_stamp: "Formatted at 2026-09-29 10:58:38 on dist-test-slave-8nrt"
server_key: "1f306cf4d7ead1e241a4fd3190dc0888"
server_key_iv: "aa43ec364b264faa99b49ec351bf6154"
server_key_version: "encryptionkey@0"
I20260929 10:58:38.942926 31492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:58:38.954135 31492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:58:38.956482 31492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:58:38.956598 31492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:58:38.960423 31492 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36689
I20260929 10:58:38.960472 31559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36689 every 8 connection(s)
I20260929 10:58:38.960845 31492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:58:38.961859 31560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:58:38.964172 31560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ff01e7d4f924fd5b0b17d0b7eb4349b: Bootstrap starting.
I20260929 10:58:38.964709 31560 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ff01e7d4f924fd5b0b17d0b7eb4349b: Neither blocks nor log segments found. Creating new log.
I20260929 10:58:38.964953 31560 log.cc:824] T 00000000000000000000000000000000 P 1ff01e7d4f924fd5b0b17d0b7eb4349b: Log is configured to *not* fsync() on all Append() calls
I20260929 10:58:38.965713 31560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ff01e7d4f924fd5b0b17d0b7eb4349b: No bootstrap required, opened a new log
I20260929 10:58:38.967308 31560 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ff01e7d4f924fd5b0b17d0b7eb4349b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ff01e7d4f924fd5b0b17d0b7eb4349b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36689 } }
I20260929 10:58:38.967448 31560 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ff01e7d4f924fd5b0b17d0b7eb4349b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:58:38.967473 31560 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ff01e7d4f924fd5b0b17d0b7eb4349b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ff01e7d4f924fd5b0b17d0b7eb4349b, State: Initialized, Role: FOLLOWER
I20260929 10:58:38.967626 31560 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ff01e7d4f924fd5b0b17d0b7eb4349b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ff01e7d4f924fd5b0b17d0b7eb4349b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36689 } }
I20260929 10:58:38.967718 31560 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ff01e7d4f924fd5b0b17d0b7eb4349b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:58:38.967747 31560 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ff01e7d4f924fd5b0b17d0b7eb4349b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:58:38.967793 31560 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ff01e7d4f924fd5b0b17d0b7eb4349b [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:58:38.968312 31560 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ff01e7d4f924fd5b0b17d0b7eb4349b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ff01e7d4f924fd5b0b17d0b7eb4349b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36689 } }
I20260929 10:58:38.968400 31560 leader_election.cc:302] T 00000000000000000000000000000000 P 1ff01e7d4f924fd5b0b17d0b7eb4349b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ff01e7d4f924fd5b0b17d0b7eb4349b; no voters:
I20260929 10:58:38.968530 31560 leader_election.cc:288] T 00000000000000000000000000000000 P 1ff01e7d4f924fd5b0b17d0b7eb4349b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:58:38.968585 31563 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ff01e7d4f924fd5b0b17d0b7eb4349b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:58:38.968779 31563 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ff01e7d4f924fd5b0b17d0b7eb4349b [term 1 LEADER]: Becoming Leader. State: Replica: 1ff01e7d4f924fd5b0b17d0b7eb4349b, State: Running, Role: LEADER
I20260929 10:58:38.968849 31560 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ff01e7d4f924fd5b0b17d0b7eb4349b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:58:38.968922 31563 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ff01e7d4f924fd5b0b17d0b7eb4349b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ff01e7d4f924fd5b0b17d0b7eb4349b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36689 } }
I20260929 10:58:38.969511 31565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ff01e7d4f924fd5b0b17d0b7eb4349b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ff01e7d4f924fd5b0b17d0b7eb4349b. Latest consensus state: current_term: 1 leader_uuid: "1ff01e7d4f924fd5b0b17d0b7eb4349b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ff01e7d4f924fd5b0b17d0b7eb4349b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36689 } } }
I20260929 10:58:38.969621 31565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ff01e7d4f924fd5b0b17d0b7eb4349b [sys.catalog]: This master's current role is: LEADER
I20260929 10:58:38.969651 31564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ff01e7d4f924fd5b0b17d0b7eb4349b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ff01e7d4f924fd5b0b17d0b7eb4349b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ff01e7d4f924fd5b0b17d0b7eb4349b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36689 } } }
I20260929 10:58:38.969740 31564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ff01e7d4f924fd5b0b17d0b7eb4349b [sys.catalog]: This master's current role is: LEADER
I20260929 10:58:38.969758 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 31492
I20260929 10:58:38.969849 31570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:58:38.969875 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790679274510869-11195-0/minicluster-data/master-0/wal/instance
I20260929 10:58:38.970136 11195 external_mini_cluster.cc:1428] Setting key 351a46defdc0fbc86b8ed71bbaf622a2
I20260929 10:58:38.970357 31570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:58:38.972246 31570 catalog_manager.cc:1347] Generated new cluster ID: 02e991083a664c4a8877728d03d034ec
I20260929 10:58:38.972304 31570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:58:38.976696 31577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:38.970804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37575 (local address 127.0.0.1:36689)
0929 10:58:38.971008 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:38.971019 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:38.971336 (+ 317us) server_negotiation.cc:407] Connection header received
0929 10:58:38.971451 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:38.971458 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:38.971512 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:38.971656 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:38.971921 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:38.973313 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:38.974443 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:38.975073 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:38.975192 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:38.975962 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:38.975966 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:38.975967 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:58:38.976179 (+ 212us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:38.976217 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:58:38.976231 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:38.976345 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:38.976439 (+ 94us) server_negotiation.cc:299] Negotiation successful
0929 10:58:38.976466 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":70,"threads_started":1}
I20260929 10:58:38.977968 11195 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:58:38.983129 31577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:38.979511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52081 (local address 127.0.0.1:36689)
0929 10:58:38.979549 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:38.979550 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:38.979710 (+ 160us) server_negotiation.cc:407] Connection header received
0929 10:58:38.979758 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:38.979760 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:38.979771 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:38.979825 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:38.979955 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:38.980696 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:38.981724 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:38.982076 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:38.982148 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:38.982873 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:38.982874 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:38.982874 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:58:38.982879 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:38.982894 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:58:38.982898 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:38.982960 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:38.983043 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 10:58:38.983056 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 10:58:38.989665 31570 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:58:38.990183 31570 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:58:39.008569 31570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ff01e7d4f924fd5b0b17d0b7eb4349b: Generated new TSK 0
I20260929 10:58:39.008790 31570 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:58:39.011207 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 31492
W20260929 10:58:39.015769 31477 connection.cc:535] client connection to 127.0.0.1:36689 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36689): Connection reset by peer (error 104)
2026-09-29T10:58:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1332 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790679274510869-11195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:58:39 dist-test-slave-8nrt krb5kdc[31591](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:58:39 dist-test-slave-8nrt krb5kdc[31591](info): set up 2 sockets
Sep 29 10:58:39 dist-test-slave-8nrt krb5kdc[31591](info): commencing operation
krb5kdc: starting...
W20260929 10:58:40.049494 11195 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Sep 29 10:58:40 dist-test-slave-8nrt krb5kdc[31591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:58:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:58:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790679274510869-11195-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790679274510869-11195-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 10:58:40.166388 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35149
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790679274510869-11195-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:58:40.248740 31604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:58:40.248939 31604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:58:40.248961 31604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:58:40.250173 31604 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:58:40.250218 31604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:58:40.250233 31604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:58:40.250245 31604 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:58:40.250257 31604 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:58:40.251825 31604 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790679274510869-11195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35149
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790679274510869-11195-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:58:40.252070 31604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:58:40.252296 31604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:58:40.257346 31609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:40.257426 31604 server_base.cc:1034] running on GCE node
W20260929 10:58:40.257349 31610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:40.257366 31612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:40.257884 31604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:58:40.258221 31604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:58:40.259395 31604 hybrid_clock.cc:648] HybridClock initialized: now 1790679520259382 us; error 42 us; skew 500 ppm
Sep 29 10:58:40 dist-test-slave-8nrt krb5kdc[31591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679520, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:58:40.265142 31604 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 10:58:40.265653 31604 webserver.cc:460] Webserver started at http://127.0.0.1:45579/ using document root <none> and password file <none>
I20260929 10:58:40.265905 31604 fs_manager.cc:359] Metadata directory not provided
I20260929 10:58:40.265978 31604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:58:40.266134 31604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:58:40.267194 31604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790679274510869-11195-0/minicluster-data/master-0/data/instance:
uuid: "875c9f3323ea49f4bf14f9d463b37ee8"
format_stamp: "Formatted at 2026-09-29 10:58:40 on dist-test-slave-8nrt"
server_key: "65dbe0fb4c51b5568becdb967e9bfb73"
server_key_iv: "d7f1736a5d9059cef192ee8433112579"
server_key_version: "encryptionkey@0"
I20260929 10:58:40.267556 31604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790679274510869-11195-0/minicluster-data/master-0/wal/instance:
uuid: "875c9f3323ea49f4bf14f9d463b37ee8"
format_stamp: "Formatted at 2026-09-29 10:58:40 on dist-test-slave-8nrt"
server_key: "65dbe0fb4c51b5568becdb967e9bfb73"
server_key_iv: "d7f1736a5d9059cef192ee8433112579"
server_key_version: "encryptionkey@0"
I20260929 10:58:40.269006 31604 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 10:58:40.269997 31619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:40.270262 31604 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 10:58:40.270391 31604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "875c9f3323ea49f4bf14f9d463b37ee8"
format_stamp: "Formatted at 2026-09-29 10:58:40 on dist-test-slave-8nrt"
server_key: "65dbe0fb4c51b5568becdb967e9bfb73"
server_key_iv: "d7f1736a5d9059cef192ee8433112579"
server_key_version: "encryptionkey@0"
I20260929 10:58:40.270488 31604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:58:40.273871 31604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:58:40.276199 31604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:58:40.276372 31604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:58:40.280323 31604 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35149
I20260929 10:58:40.280373 31671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35149 every 8 connection(s)
I20260929 10:58:40.280807 31604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:58:40.281206 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 31604
I20260929 10:58:40.281348 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790679274510869-11195-0/minicluster-data/master-0/wal/instance
I20260929 10:58:40.281554 11195 external_mini_cluster.cc:1428] Setting key 4ff1cad1667b9f7ca1c6f1bc54b1d159
I20260929 10:58:40.282064 31672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:58:40.284365 31675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:40.282192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49981 (local address 127.0.0.1:35149)
0929 10:58:40.282400 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:40.282414 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:40.282438 (+ 24us) server_negotiation.cc:407] Connection header received
0929 10:58:40.283101 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:40.283107 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:40.283132 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:40.283283 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:40.283942 (+ 659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:40.283945 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:40.283946 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:58:40.283950 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:40.283998 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:58:40.284009 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:40.284083 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:40.284165 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 10:58:40.284207 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":89,"threads_started":1}
I20260929 10:58:40.285312 31672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 875c9f3323ea49f4bf14f9d463b37ee8: Bootstrap starting.
I20260929 10:58:40.285511 11195 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:58:40.285970 31672 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 875c9f3323ea49f4bf14f9d463b37ee8: Neither blocks nor log segments found. Creating new log.
I20260929 10:58:40.286262 31672 log.cc:824] T 00000000000000000000000000000000 P 875c9f3323ea49f4bf14f9d463b37ee8: Log is configured to *not* fsync() on all Append() calls
I20260929 10:58:40.287060 31672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 875c9f3323ea49f4bf14f9d463b37ee8: No bootstrap required, opened a new log
I20260929 10:58:40.288036 31675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:40.286989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53875 (local address 127.0.0.1:35149)
0929 10:58:40.287025 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:40.287026 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:40.287167 (+ 141us) server_negotiation.cc:407] Connection header received
0929 10:58:40.287239 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:40.287240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:40.287245 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:40.287300 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:40.287842 (+ 542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:40.287843 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:40.287844 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:58:40.287848 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:40.287861 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:58:40.287865 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:40.287909 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:40.287961 (+ 52us) server_negotiation.cc:299] Negotiation successful
0929 10:58:40.287975 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 10:58:40.289340 31672 raft_consensus.cc:348] T 00000000000000000000000000000000 P 875c9f3323ea49f4bf14f9d463b37ee8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "875c9f3323ea49f4bf14f9d463b37ee8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35149 } }
I20260929 10:58:40.289479 31672 raft_consensus.cc:374] T 00000000000000000000000000000000 P 875c9f3323ea49f4bf14f9d463b37ee8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:58:40.289501 31672 raft_consensus.cc:729] T 00000000000000000000000000000000 P 875c9f3323ea49f4bf14f9d463b37ee8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 875c9f3323ea49f4bf14f9d463b37ee8, State: Initialized, Role: FOLLOWER
I20260929 10:58:40.289611 31672 consensus_queue.cc:260] T 00000000000000000000000000000000 P 875c9f3323ea49f4bf14f9d463b37ee8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "875c9f3323ea49f4bf14f9d463b37ee8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35149 } }
I20260929 10:58:40.289690 31672 raft_consensus.cc:388] T 00000000000000000000000000000000 P 875c9f3323ea49f4bf14f9d463b37ee8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:58:40.289712 31672 raft_consensus.cc:482] T 00000000000000000000000000000000 P 875c9f3323ea49f4bf14f9d463b37ee8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:58:40.289750 31672 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 875c9f3323ea49f4bf14f9d463b37ee8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:58:40.290235 31672 raft_consensus.cc:504] T 00000000000000000000000000000000 P 875c9f3323ea49f4bf14f9d463b37ee8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "875c9f3323ea49f4bf14f9d463b37ee8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35149 } }
I20260929 10:58:40.290321 31672 leader_election.cc:302] T 00000000000000000000000000000000 P 875c9f3323ea49f4bf14f9d463b37ee8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 875c9f3323ea49f4bf14f9d463b37ee8; no voters:
I20260929 10:58:40.290447 31672 leader_election.cc:288] T 00000000000000000000000000000000 P 875c9f3323ea49f4bf14f9d463b37ee8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:58:40.290542 31682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 875c9f3323ea49f4bf14f9d463b37ee8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:58:40.290675 31672 sys_catalog.cc:564] T 00000000000000000000000000000000 P 875c9f3323ea49f4bf14f9d463b37ee8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:58:40.290699 31682 raft_consensus.cc:686] T 00000000000000000000000000000000 P 875c9f3323ea49f4bf14f9d463b37ee8 [term 1 LEADER]: Becoming Leader. State: Replica: 875c9f3323ea49f4bf14f9d463b37ee8, State: Running, Role: LEADER
I20260929 10:58:40.290820 31682 consensus_queue.cc:237] T 00000000000000000000000000000000 P 875c9f3323ea49f4bf14f9d463b37ee8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "875c9f3323ea49f4bf14f9d463b37ee8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35149 } }
I20260929 10:58:40.291347 31683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 875c9f3323ea49f4bf14f9d463b37ee8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "875c9f3323ea49f4bf14f9d463b37ee8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "875c9f3323ea49f4bf14f9d463b37ee8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35149 } } }
I20260929 10:58:40.291436 31683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 875c9f3323ea49f4bf14f9d463b37ee8 [sys.catalog]: This master's current role is: LEADER
I20260929 10:58:40.291324 31684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 875c9f3323ea49f4bf14f9d463b37ee8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 875c9f3323ea49f4bf14f9d463b37ee8. Latest consensus state: current_term: 1 leader_uuid: "875c9f3323ea49f4bf14f9d463b37ee8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "875c9f3323ea49f4bf14f9d463b37ee8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35149 } } }
I20260929 10:58:40.291594 31684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 875c9f3323ea49f4bf14f9d463b37ee8 [sys.catalog]: This master's current role is: LEADER
I20260929 10:58:40.291663 31691 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:58:40.292150 31691 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:58:40.293727 31691 catalog_manager.cc:1347] Generated new cluster ID: ca7bd688933d4115a2f4678a1fcb611b
I20260929 10:58:40.293771 31691 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:58:40.296808 31691 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:58:40.296960 31691 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:58:40.299708 31691 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 875c9f3323ea49f4bf14f9d463b37ee8: Generated new TSK 0
I20260929 10:58:40.299883 31691 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:58:40.306511 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 31604
2026-09-29T10:58:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1294 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790679274510869-11195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:58:40 dist-test-slave-8nrt krb5kdc[31703](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:58:40 dist-test-slave-8nrt krb5kdc[31703](info): set up 2 sockets
Sep 29 10:58:40 dist-test-slave-8nrt krb5kdc[31703](info): commencing operation
krb5kdc: starting...
W20260929 10:58:41.349376 11195 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Sep 29 10:58:41 dist-test-slave-8nrt krb5kdc[31703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:58:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:58:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790679274510869-11195-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790679274510869-11195-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 10:58:41.466012 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36947
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790679274510869-11195-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:58:41.546695 31716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:58:41.546898 31716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:58:41.546921 31716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:58:41.548106 31716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:58:41.548151 31716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:58:41.548166 31716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:58:41.548178 31716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:58:41.548190 31716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:58:41.549700 31716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790679274510869-11195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790679274510869-11195-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:58:41.549957 31716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:58:41.550222 31716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:58:41.555075 31721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:41.555114 31724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:41.555152 31722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:41.555234 31716 server_base.cc:1034] running on GCE node
I20260929 10:58:41.555497 31716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:58:41.555778 31716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:58:41.556918 31716 hybrid_clock.cc:648] HybridClock initialized: now 1790679521556907 us; error 38 us; skew 500 ppm
Sep 29 10:58:41 dist-test-slave-8nrt krb5kdc[31703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679521, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:58:41.559429 31716 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 10:58:41.559804 31716 webserver.cc:460] Webserver started at http://127.0.0.1:38771/ using document root <none> and password file <none>
I20260929 10:58:41.560046 31716 fs_manager.cc:359] Metadata directory not provided
I20260929 10:58:41.560119 31716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:58:41.560281 31716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:58:41.561417 31716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790679274510869-11195-0/minicluster-data/master-0/data/instance:
uuid: "db6ab257b18b41488707b3f08f210dee"
format_stamp: "Formatted at 2026-09-29 10:58:41 on dist-test-slave-8nrt"
server_key: "69f0b12c7b17ec05b119d4a1274d1b61"
server_key_iv: "fa49f9031576dde44b6209384a2a7bf8"
server_key_version: "encryptionkey@0"
I20260929 10:58:41.561794 31716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790679274510869-11195-0/minicluster-data/master-0/wal/instance:
uuid: "db6ab257b18b41488707b3f08f210dee"
format_stamp: "Formatted at 2026-09-29 10:58:41 on dist-test-slave-8nrt"
server_key: "69f0b12c7b17ec05b119d4a1274d1b61"
server_key_iv: "fa49f9031576dde44b6209384a2a7bf8"
server_key_version: "encryptionkey@0"
I20260929 10:58:41.563088 31716 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:58:41.563818 31731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:41.563990 31716 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:58:41.564095 31716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "db6ab257b18b41488707b3f08f210dee"
format_stamp: "Formatted at 2026-09-29 10:58:41 on dist-test-slave-8nrt"
server_key: "69f0b12c7b17ec05b119d4a1274d1b61"
server_key_iv: "fa49f9031576dde44b6209384a2a7bf8"
server_key_version: "encryptionkey@0"
I20260929 10:58:41.564175 31716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:58:41.576411 31716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:58:41.578872 31716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:58:41.578989 31716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:58:41.582863 31716 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36947
I20260929 10:58:41.582896 31783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36947 every 8 connection(s)
I20260929 10:58:41.583266 31716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:58:41.584211 31784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:58:41.586506 31784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db6ab257b18b41488707b3f08f210dee: Bootstrap starting.
I20260929 10:58:41.587085 31784 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db6ab257b18b41488707b3f08f210dee: Neither blocks nor log segments found. Creating new log.
I20260929 10:58:41.587378 31784 log.cc:824] T 00000000000000000000000000000000 P db6ab257b18b41488707b3f08f210dee: Log is configured to *not* fsync() on all Append() calls
I20260929 10:58:41.588058 31784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db6ab257b18b41488707b3f08f210dee: No bootstrap required, opened a new log
I20260929 10:58:41.589736 31784 raft_consensus.cc:348] T 00000000000000000000000000000000 P db6ab257b18b41488707b3f08f210dee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db6ab257b18b41488707b3f08f210dee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36947 } }
I20260929 10:58:41.589874 31784 raft_consensus.cc:374] T 00000000000000000000000000000000 P db6ab257b18b41488707b3f08f210dee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:58:41.589896 31784 raft_consensus.cc:729] T 00000000000000000000000000000000 P db6ab257b18b41488707b3f08f210dee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db6ab257b18b41488707b3f08f210dee, State: Initialized, Role: FOLLOWER
I20260929 10:58:41.590049 31784 consensus_queue.cc:260] T 00000000000000000000000000000000 P db6ab257b18b41488707b3f08f210dee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db6ab257b18b41488707b3f08f210dee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36947 } }
I20260929 10:58:41.590117 31784 raft_consensus.cc:388] T 00000000000000000000000000000000 P db6ab257b18b41488707b3f08f210dee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:58:41.590139 31784 raft_consensus.cc:482] T 00000000000000000000000000000000 P db6ab257b18b41488707b3f08f210dee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:58:41.590214 31784 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db6ab257b18b41488707b3f08f210dee [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:58:41.590740 31784 raft_consensus.cc:504] T 00000000000000000000000000000000 P db6ab257b18b41488707b3f08f210dee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db6ab257b18b41488707b3f08f210dee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36947 } }
I20260929 10:58:41.590852 31784 leader_election.cc:302] T 00000000000000000000000000000000 P db6ab257b18b41488707b3f08f210dee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db6ab257b18b41488707b3f08f210dee; no voters:
I20260929 10:58:41.591028 31784 leader_election.cc:288] T 00000000000000000000000000000000 P db6ab257b18b41488707b3f08f210dee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:58:41.591046 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 31716
I20260929 10:58:41.591111 31787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db6ab257b18b41488707b3f08f210dee [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:58:41.591157 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790679274510869-11195-0/minicluster-data/master-0/wal/instance
I20260929 10:58:41.591301 31784 sys_catalog.cc:564] T 00000000000000000000000000000000 P db6ab257b18b41488707b3f08f210dee [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:58:41.591351 11195 external_mini_cluster.cc:1428] Setting key 43da9b06513dc62f9b33fe8b0d67314b
I20260929 10:58:41.591341 31787 raft_consensus.cc:686] T 00000000000000000000000000000000 P db6ab257b18b41488707b3f08f210dee [term 1 LEADER]: Becoming Leader. State: Replica: db6ab257b18b41488707b3f08f210dee, State: Running, Role: LEADER
I20260929 10:58:41.591470 31787 consensus_queue.cc:237] T 00000000000000000000000000000000 P db6ab257b18b41488707b3f08f210dee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db6ab257b18b41488707b3f08f210dee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36947 } }
I20260929 10:58:41.591796 31788 sys_catalog.cc:455] T 00000000000000000000000000000000 P db6ab257b18b41488707b3f08f210dee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db6ab257b18b41488707b3f08f210dee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db6ab257b18b41488707b3f08f210dee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36947 } } }
I20260929 10:58:41.591905 31788 sys_catalog.cc:458] T 00000000000000000000000000000000 P db6ab257b18b41488707b3f08f210dee [sys.catalog]: This master's current role is: LEADER
I20260929 10:58:41.591827 31789 sys_catalog.cc:455] T 00000000000000000000000000000000 P db6ab257b18b41488707b3f08f210dee [sys.catalog]: SysCatalogTable state changed. Reason: New leader db6ab257b18b41488707b3f08f210dee. Latest consensus state: current_term: 1 leader_uuid: "db6ab257b18b41488707b3f08f210dee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db6ab257b18b41488707b3f08f210dee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36947 } } }
I20260929 10:58:41.592132 31789 sys_catalog.cc:458] T 00000000000000000000000000000000 P db6ab257b18b41488707b3f08f210dee [sys.catalog]: This master's current role is: LEADER
I20260929 10:58:41.592294 31793 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:58:41.592916 31793 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:58:41.594676 31793 catalog_manager.cc:1347] Generated new cluster ID: 7b95d7759cd943d9981e213bd101a46b
I20260929 10:58:41.594728 31793 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 10:58:41 dist-test-slave-8nrt krb5kdc[31703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 10:58:41.603257 31793 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:58:41.603837 31793 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:58:41.604916 31792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:41.591854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44375 (local address 127.0.0.1:36947)
0929 10:58:41.592422 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:41.592432 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:41.592468 (+ 36us) server_negotiation.cc:407] Connection header received
0929 10:58:41.592637 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:41.592644 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:41.592708 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:41.593735 (+ 1027us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:41.593875 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:41.595233 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:41.596294 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:41.596904 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:41.596999 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:41.599914 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:41.599918 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:41.599919 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:58:41.599931 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:41.602684 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:41.603127 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:41.603130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:41.603131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:41.603265 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:41.603544 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:41.603545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:41.603545 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:41.603891 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:41.604055 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:41.604525 (+ 470us) server_negotiation.cc:299] Negotiation successful
0929 10:58:41.604688 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":513,"thread_start_us":70,"threads_started":1}
I20260929 10:58:41.605958 11195 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:58:41.609128 31793 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db6ab257b18b41488707b3f08f210dee: Generated new TSK 0
I20260929 10:58:41.609360 31793 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:58:41.614279 31792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:41.607506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36477 (local address 127.0.0.1:36947)
0929 10:58:41.607545 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:41.607546 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:41.607586 (+ 40us) server_negotiation.cc:407] Connection header received
0929 10:58:41.607643 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:41.607644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:41.607654 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:41.607728 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:41.608313 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:41.609120 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:41.609962 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:41.610267 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:41.610339 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:41.611541 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:41.611542 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:41.611542 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:58:41.611549 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:41.613117 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:41.613529 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:41.613530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:41.613530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:41.613601 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:41.613791 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:41.613792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:41.613792 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:41.613919 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:41.614021 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:41.614098 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 10:58:41.614179 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 10:58:41.616432 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 31716
2026-09-29T10:58:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1309 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790679274510869-11195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:58:41 dist-test-slave-8nrt krb5kdc[31815](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:58:41 dist-test-slave-8nrt krb5kdc[31815](info): set up 2 sockets
Sep 29 10:58:41 dist-test-slave-8nrt krb5kdc[31815](info): commencing operation
krb5kdc: starting...
W20260929 10:58:42.649403 11195 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Sep 29 10:58:42 dist-test-slave-8nrt krb5kdc[31815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:58:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:58:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790679274510869-11195-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790679274510869-11195-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 10:58:42.768389 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42841
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790679274510869-11195-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42841
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:58:42.866945 31828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:58:42.867177 31828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:58:42.867218 31828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:58:42.868402 31828 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:58:42.868474 31828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:58:42.868499 31828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:58:42.868530 31828 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:58:42.868552 31828 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:58:42.870083 31828 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790679274510869-11195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790679274510869-11195-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:58:42.870348 31828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:58:42.870573 31828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:58:42.875420 31836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:42.875438 31834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:42.875433 31833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:42.875797 31828 server_base.cc:1034] running on GCE node
I20260929 10:58:42.876012 31828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:58:42.876246 31828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:58:42.877419 31828 hybrid_clock.cc:648] HybridClock initialized: now 1790679522877400 us; error 29 us; skew 500 ppm
Sep 29 10:58:42 dist-test-slave-8nrt krb5kdc[31815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:58:42.879809 31828 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 10:58:42.880337 31828 webserver.cc:460] Webserver started at http://127.0.0.1:34865/ using document root <none> and password file <none>
I20260929 10:58:42.880584 31828 fs_manager.cc:359] Metadata directory not provided
I20260929 10:58:42.880653 31828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:58:42.880813 31828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:58:42.881927 31828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790679274510869-11195-0/minicluster-data/master-0/data/instance:
uuid: "7d7122dda92a46a4b973d1d3c40d4194"
format_stamp: "Formatted at 2026-09-29 10:58:42 on dist-test-slave-8nrt"
server_key: "4f922a2b0dfb56e47bb9df48ee7e616c"
server_key_iv: "fec8649e1ad3839d8fd2042a18f7e27c"
server_key_version: "encryptionkey@0"
I20260929 10:58:42.882305 31828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790679274510869-11195-0/minicluster-data/master-0/wal/instance:
uuid: "7d7122dda92a46a4b973d1d3c40d4194"
format_stamp: "Formatted at 2026-09-29 10:58:42 on dist-test-slave-8nrt"
server_key: "4f922a2b0dfb56e47bb9df48ee7e616c"
server_key_iv: "fec8649e1ad3839d8fd2042a18f7e27c"
server_key_version: "encryptionkey@0"
I20260929 10:58:42.883605 31828 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 10:58:42.884313 31843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:42.884482 31828 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:58:42.884582 31828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "7d7122dda92a46a4b973d1d3c40d4194"
format_stamp: "Formatted at 2026-09-29 10:58:42 on dist-test-slave-8nrt"
server_key: "4f922a2b0dfb56e47bb9df48ee7e616c"
server_key_iv: "fec8649e1ad3839d8fd2042a18f7e27c"
server_key_version: "encryptionkey@0"
I20260929 10:58:42.884665 31828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:58:42.906896 31828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:58:42.908722 31828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:58:42.908895 31828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:58:42.913157 31828 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42841
I20260929 10:58:42.913206 31895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42841 every 8 connection(s)
I20260929 10:58:42.913620 31828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:58:42.914469 31896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:58:42.915023 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 31828
I20260929 10:58:42.915148 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790679274510869-11195-0/minicluster-data/master-0/wal/instance
I20260929 10:58:42.915360 11195 external_mini_cluster.cc:1428] Setting key 65b8000127d17cce5193f562c4544b46
I20260929 10:58:42.917533 31896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d7122dda92a46a4b973d1d3c40d4194: Bootstrap starting.
I20260929 10:58:42.918190 31896 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d7122dda92a46a4b973d1d3c40d4194: Neither blocks nor log segments found. Creating new log.
I20260929 10:58:42.918504 31896 log.cc:824] T 00000000000000000000000000000000 P 7d7122dda92a46a4b973d1d3c40d4194: Log is configured to *not* fsync() on all Append() calls
I20260929 10:58:42.919224 31896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d7122dda92a46a4b973d1d3c40d4194: No bootstrap required, opened a new log
I20260929 10:58:42.920930 31899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:42.915915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50153 (local address 127.0.0.1:42841)
0929 10:58:42.916089 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:42.916097 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:42.916122 (+ 25us) server_negotiation.cc:407] Connection header received
0929 10:58:42.916551 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:42.916557 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:42.916605 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:42.916717 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:42.916969 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:42.917969 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:42.919082 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:42.919523 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:42.919603 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:42.920287 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:42.920291 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:42.920291 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:58:42.920444 (+ 153us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:42.920473 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:58:42.920483 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:42.920535 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:42.920642 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 10:58:42.920687 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":77,"threads_started":1}
I20260929 10:58:42.921578 31896 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d7122dda92a46a4b973d1d3c40d4194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d7122dda92a46a4b973d1d3c40d4194" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42841 } }
I20260929 10:58:42.921795 31896 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d7122dda92a46a4b973d1d3c40d4194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:58:42.921846 31896 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d7122dda92a46a4b973d1d3c40d4194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d7122dda92a46a4b973d1d3c40d4194, State: Initialized, Role: FOLLOWER
I20260929 10:58:42.921972 11195 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:58:42.921977 31896 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d7122dda92a46a4b973d1d3c40d4194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d7122dda92a46a4b973d1d3c40d4194" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42841 } }
I20260929 10:58:42.922060 31896 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d7122dda92a46a4b973d1d3c40d4194 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:58:42.922096 31896 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d7122dda92a46a4b973d1d3c40d4194 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:58:42.922137 31896 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d7122dda92a46a4b973d1d3c40d4194 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:58:42.922827 31896 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d7122dda92a46a4b973d1d3c40d4194 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d7122dda92a46a4b973d1d3c40d4194" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42841 } }
I20260929 10:58:42.922940 31896 leader_election.cc:302] T 00000000000000000000000000000000 P 7d7122dda92a46a4b973d1d3c40d4194 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d7122dda92a46a4b973d1d3c40d4194; no voters:
I20260929 10:58:42.923103 31896 leader_election.cc:288] T 00000000000000000000000000000000 P 7d7122dda92a46a4b973d1d3c40d4194 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:58:42.923141 31903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d7122dda92a46a4b973d1d3c40d4194 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:58:42.923354 31903 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d7122dda92a46a4b973d1d3c40d4194 [term 1 LEADER]: Becoming Leader. State: Replica: 7d7122dda92a46a4b973d1d3c40d4194, State: Running, Role: LEADER
I20260929 10:58:42.923426 31896 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d7122dda92a46a4b973d1d3c40d4194 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:58:42.923571 31903 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d7122dda92a46a4b973d1d3c40d4194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d7122dda92a46a4b973d1d3c40d4194" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42841 } }
I20260929 10:58:42.923975 31906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d7122dda92a46a4b973d1d3c40d4194 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d7122dda92a46a4b973d1d3c40d4194. Latest consensus state: current_term: 1 leader_uuid: "7d7122dda92a46a4b973d1d3c40d4194" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d7122dda92a46a4b973d1d3c40d4194" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42841 } } }
I20260929 10:58:42.924054 31906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d7122dda92a46a4b973d1d3c40d4194 [sys.catalog]: This master's current role is: LEADER
I20260929 10:58:42.924266 31914 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:58:42.924271 31907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d7122dda92a46a4b973d1d3c40d4194 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d7122dda92a46a4b973d1d3c40d4194" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d7122dda92a46a4b973d1d3c40d4194" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42841 } } }
I20260929 10:58:42.924686 31907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d7122dda92a46a4b973d1d3c40d4194 [sys.catalog]: This master's current role is: LEADER
I20260929 10:58:42.925120 31914 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:58:42.926745 31914 catalog_manager.cc:1347] Generated new cluster ID: efc19ee5e6c84845b8c4e78aad5fb8d7
I20260929 10:58:42.926793 31914 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:58:42.928249 31899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:42.923473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35343 (local address 127.0.0.1:42841)
0929 10:58:42.923535 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:42.923537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:42.924914 (+ 1377us) server_negotiation.cc:407] Connection header received
0929 10:58:42.924943 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:42.924944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:42.924954 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:42.925011 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:42.925205 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:42.925891 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:42.927044 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:42.927254 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:42.927339 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:42.928008 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:42.928009 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:42.928010 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:58:42.928014 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:42.928027 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:58:42.928031 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:42.928079 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:42.928133 (+ 54us) server_negotiation.cc:299] Negotiation successful
0929 10:58:42.928153 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 10:58:42.953595 31914 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:58:42.954197 31914 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:58:42.956372 31914 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d7122dda92a46a4b973d1d3c40d4194: Generated new TSK 0
I20260929 10:58:42.956566 31914 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:58:42.967772 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 31828
2026-09-29T10:58:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1351 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790679274510869-11195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:58:43 dist-test-slave-8nrt krb5kdc[31927](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:58:43 dist-test-slave-8nrt krb5kdc[31927](info): set up 2 sockets
Sep 29 10:58:43 dist-test-slave-8nrt krb5kdc[31927](info): commencing operation
krb5kdc: starting...
W20260929 10:58:43.993340 11195 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Sep 29 10:58:44 dist-test-slave-8nrt krb5kdc[31927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:58:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:58:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790679274510869-11195-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790679274510869-11195-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 10:58:44.111673 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42931
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790679274510869-11195-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42931
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:58:44.191790 31940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:58:44.191991 31940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:58:44.192013 31940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:58:44.193163 31940 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:58:44.193207 31940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:58:44.193220 31940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:58:44.193233 31940 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:58:44.193305 31940 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:58:44.194795 31940 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790679274510869-11195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790679274510869-11195-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:58:44.195037 31940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:58:44.195258 31940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 10:58:44.200155 31940 server_base.cc:1034] running on GCE node
W20260929 10:58:44.200160 31946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:44.200291 31948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:44.200263 31945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:44.200605 31940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:58:44.200881 31940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:58:44.202046 31940 hybrid_clock.cc:648] HybridClock initialized: now 1790679524202024 us; error 51 us; skew 500 ppm
Sep 29 10:58:44 dist-test-slave-8nrt krb5kdc[31927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679524, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:58:44.204516 31940 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 10:58:44.204991 31940 webserver.cc:460] Webserver started at http://127.0.0.1:40229/ using document root <none> and password file <none>
I20260929 10:58:44.205219 31940 fs_manager.cc:359] Metadata directory not provided
I20260929 10:58:44.205312 31940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:58:44.205466 31940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:58:44.206535 31940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790679274510869-11195-0/minicluster-data/master-0/data/instance:
uuid: "a1e9f5af84f044a19486a4fa8b74e74e"
format_stamp: "Formatted at 2026-09-29 10:58:44 on dist-test-slave-8nrt"
server_key: "d9f8728fda7f75b450dc9a161b7fda22"
server_key_iv: "b35b836f2ecd91388508c39b0bd4bd9b"
server_key_version: "encryptionkey@0"
I20260929 10:58:44.206895 31940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790679274510869-11195-0/minicluster-data/master-0/wal/instance:
uuid: "a1e9f5af84f044a19486a4fa8b74e74e"
format_stamp: "Formatted at 2026-09-29 10:58:44 on dist-test-slave-8nrt"
server_key: "d9f8728fda7f75b450dc9a161b7fda22"
server_key_iv: "b35b836f2ecd91388508c39b0bd4bd9b"
server_key_version: "encryptionkey@0"
I20260929 10:58:44.208173 31940 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 10:58:44.208851 31955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:44.209021 31940 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 10:58:44.209133 31940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "a1e9f5af84f044a19486a4fa8b74e74e"
format_stamp: "Formatted at 2026-09-29 10:58:44 on dist-test-slave-8nrt"
server_key: "d9f8728fda7f75b450dc9a161b7fda22"
server_key_iv: "b35b836f2ecd91388508c39b0bd4bd9b"
server_key_version: "encryptionkey@0"
I20260929 10:58:44.209208 31940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:58:44.212433 31940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:58:44.214756 31940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:58:44.214913 31940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:58:44.219007 31940 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42931
I20260929 10:58:44.219034 32007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42931 every 8 connection(s)
I20260929 10:58:44.219357 31940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:58:44.220284 32008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:58:44.222399 32008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1e9f5af84f044a19486a4fa8b74e74e: Bootstrap starting.
I20260929 10:58:44.222891 32008 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1e9f5af84f044a19486a4fa8b74e74e: Neither blocks nor log segments found. Creating new log.
I20260929 10:58:44.223125 32008 log.cc:824] T 00000000000000000000000000000000 P a1e9f5af84f044a19486a4fa8b74e74e: Log is configured to *not* fsync() on all Append() calls
I20260929 10:58:44.223754 32008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1e9f5af84f044a19486a4fa8b74e74e: No bootstrap required, opened a new log
I20260929 10:58:44.225482 32008 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1e9f5af84f044a19486a4fa8b74e74e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1e9f5af84f044a19486a4fa8b74e74e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42931 } }
I20260929 10:58:44.225610 32008 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1e9f5af84f044a19486a4fa8b74e74e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:58:44.225632 32008 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1e9f5af84f044a19486a4fa8b74e74e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1e9f5af84f044a19486a4fa8b74e74e, State: Initialized, Role: FOLLOWER
I20260929 10:58:44.225755 32008 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1e9f5af84f044a19486a4fa8b74e74e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1e9f5af84f044a19486a4fa8b74e74e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42931 } }
I20260929 10:58:44.225834 32008 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1e9f5af84f044a19486a4fa8b74e74e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:58:44.225857 32008 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1e9f5af84f044a19486a4fa8b74e74e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:58:44.225883 32008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1e9f5af84f044a19486a4fa8b74e74e [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:58:44.226297 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 31940
I20260929 10:58:44.226353 32008 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1e9f5af84f044a19486a4fa8b74e74e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1e9f5af84f044a19486a4fa8b74e74e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42931 } }
I20260929 10:58:44.226446 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790679274510869-11195-0/minicluster-data/master-0/wal/instance
I20260929 10:58:44.226475 32008 leader_election.cc:302] T 00000000000000000000000000000000 P a1e9f5af84f044a19486a4fa8b74e74e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1e9f5af84f044a19486a4fa8b74e74e; no voters:
I20260929 10:58:44.226645 32008 leader_election.cc:288] T 00000000000000000000000000000000 P a1e9f5af84f044a19486a4fa8b74e74e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:58:44.226681 11195 external_mini_cluster.cc:1428] Setting key f3d258a5f0555f9e7af6b03c3155f008
I20260929 10:58:44.226727 32011 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1e9f5af84f044a19486a4fa8b74e74e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:58:44.226893 32008 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1e9f5af84f044a19486a4fa8b74e74e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:58:44.226944 32011 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1e9f5af84f044a19486a4fa8b74e74e [term 1 LEADER]: Becoming Leader. State: Replica: a1e9f5af84f044a19486a4fa8b74e74e, State: Running, Role: LEADER
I20260929 10:58:44.227080 32011 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1e9f5af84f044a19486a4fa8b74e74e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1e9f5af84f044a19486a4fa8b74e74e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42931 } }
I20260929 10:58:44.227523 32013 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1e9f5af84f044a19486a4fa8b74e74e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1e9f5af84f044a19486a4fa8b74e74e. Latest consensus state: current_term: 1 leader_uuid: "a1e9f5af84f044a19486a4fa8b74e74e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1e9f5af84f044a19486a4fa8b74e74e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42931 } } }
I20260929 10:58:44.227613 32013 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1e9f5af84f044a19486a4fa8b74e74e [sys.catalog]: This master's current role is: LEADER
I20260929 10:58:44.227505 32012 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1e9f5af84f044a19486a4fa8b74e74e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1e9f5af84f044a19486a4fa8b74e74e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1e9f5af84f044a19486a4fa8b74e74e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42931 } } }
I20260929 10:58:44.227798 32012 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1e9f5af84f044a19486a4fa8b74e74e [sys.catalog]: This master's current role is: LEADER
I20260929 10:58:44.228169 32019 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:58:44.228663 32019 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:58:44.229538 32015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:44.227307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36259 (local address 127.0.0.1:42931)
0929 10:58:44.227475 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:44.227484 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:44.227695 (+ 211us) server_negotiation.cc:407] Connection header received
0929 10:58:44.227842 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:44.227851 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:44.227886 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:44.228028 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:44.229046 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:44.229050 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:44.229050 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:58:44.229055 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:44.229097 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:58:44.229113 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:44.229175 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:44.229330 (+ 155us) server_negotiation.cc:299] Negotiation successful
0929 10:58:44.229371 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":65,"threads_started":1}
I20260929 10:58:44.230491 32019 catalog_manager.cc:1347] Generated new cluster ID: 84be21c6ece0429d8cfb37b079a5ec0b
I20260929 10:58:44.230541 32019 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:58:44.230562 11195 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:58:44.233114 32015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:44.231973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47675 (local address 127.0.0.1:42931)
0929 10:58:44.232007 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:44.232008 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:44.232152 (+ 144us) server_negotiation.cc:407] Connection header received
0929 10:58:44.232230 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:44.232231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:44.232246 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:44.232294 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:44.232896 (+ 602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:44.232897 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:44.232898 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:58:44.232902 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:44.232915 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:58:44.232919 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:44.232966 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:44.233019 (+ 53us) server_negotiation.cc:299] Negotiation successful
0929 10:58:44.233033 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 10:58:44.245630 32019 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:58:44.245819 32019 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:58:44.249720 32019 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1e9f5af84f044a19486a4fa8b74e74e: Generated new TSK 0
I20260929 10:58:44.249904 32019 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:58:44.255479 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 31940
2026-09-29T10:58:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1287 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790679274510869-11195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:58:44 dist-test-slave-8nrt krb5kdc[32039](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:58:44 dist-test-slave-8nrt krb5kdc[32039](info): set up 2 sockets
Sep 29 10:58:44 dist-test-slave-8nrt krb5kdc[32039](info): commencing operation
krb5kdc: starting...
W20260929 10:58:45.289425 11195 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Sep 29 10:58:45 dist-test-slave-8nrt krb5kdc[32039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:58:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:58:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790679274510869-11195-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790679274510869-11195-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 10:58:45.403908 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45315
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790679274510869-11195-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:58:45.483222 32052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:58:45.483418 32052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:58:45.483440 32052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:58:45.484591 32052 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:58:45.484642 32052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:58:45.484656 32052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:58:45.484669 32052 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:58:45.484681 32052 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:58:45.486171 32052 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790679274510869-11195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790679274510869-11195-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:58:45.486426 32052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:58:45.486662 32052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:58:45.491500 32058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:45.491474 32057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:45.491775 32052 server_base.cc:1034] running on GCE node
W20260929 10:58:45.491482 32060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:45.492072 32052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:58:45.492442 32052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:58:45.493602 32052 hybrid_clock.cc:648] HybridClock initialized: now 1790679525493588 us; error 26 us; skew 500 ppm
Sep 29 10:58:45 dist-test-slave-8nrt krb5kdc[32039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679525, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:58:45.500460 32052 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 10:58:45.500880 32052 webserver.cc:460] Webserver started at http://127.0.0.1:37203/ using document root <none> and password file <none>
I20260929 10:58:45.501117 32052 fs_manager.cc:359] Metadata directory not provided
I20260929 10:58:45.501195 32052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:58:45.501370 32052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:58:45.502429 32052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790679274510869-11195-0/minicluster-data/master-0/data/instance:
uuid: "49227c63d1f4450084d265a1e2a7789b"
format_stamp: "Formatted at 2026-09-29 10:58:45 on dist-test-slave-8nrt"
server_key: "3e3e728fb0916409b37360ddfbb1125c"
server_key_iv: "39c065fa96c345cf6577cac92cd6f07e"
server_key_version: "encryptionkey@0"
I20260929 10:58:45.502799 32052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790679274510869-11195-0/minicluster-data/master-0/wal/instance:
uuid: "49227c63d1f4450084d265a1e2a7789b"
format_stamp: "Formatted at 2026-09-29 10:58:45 on dist-test-slave-8nrt"
server_key: "3e3e728fb0916409b37360ddfbb1125c"
server_key_iv: "39c065fa96c345cf6577cac92cd6f07e"
server_key_version: "encryptionkey@0"
I20260929 10:58:45.504096 32052 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 10:58:45.504806 32067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:45.504976 32052 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:58:45.505077 32052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "49227c63d1f4450084d265a1e2a7789b"
format_stamp: "Formatted at 2026-09-29 10:58:45 on dist-test-slave-8nrt"
server_key: "3e3e728fb0916409b37360ddfbb1125c"
server_key_iv: "39c065fa96c345cf6577cac92cd6f07e"
server_key_version: "encryptionkey@0"
I20260929 10:58:45.505146 32052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:58:45.518615 32052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:58:45.520426 32052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:58:45.520586 32052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:58:45.524605 32052 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45315
I20260929 10:58:45.524662 32119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45315 every 8 connection(s)
I20260929 10:58:45.525017 32052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:58:45.525838 32120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:58:45.528028 32120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49227c63d1f4450084d265a1e2a7789b: Bootstrap starting.
I20260929 10:58:45.528299 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 32052
I20260929 10:58:45.528435 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790679274510869-11195-0/minicluster-data/master-0/wal/instance
I20260929 10:58:45.528630 11195 external_mini_cluster.cc:1428] Setting key 141458a59abb4e2399594af7d19b3876
I20260929 10:58:45.528683 32120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49227c63d1f4450084d265a1e2a7789b: Neither blocks nor log segments found. Creating new log.
I20260929 10:58:45.529062 32120 log.cc:824] T 00000000000000000000000000000000 P 49227c63d1f4450084d265a1e2a7789b: Log is configured to *not* fsync() on all Append() calls
I20260929 10:58:45.529810 32120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49227c63d1f4450084d265a1e2a7789b: No bootstrap required, opened a new log
I20260929 10:58:45.532130 32120 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49227c63d1f4450084d265a1e2a7789b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49227c63d1f4450084d265a1e2a7789b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45315 } }
I20260929 10:58:45.532303 32120 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49227c63d1f4450084d265a1e2a7789b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:58:45.532339 32120 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49227c63d1f4450084d265a1e2a7789b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49227c63d1f4450084d265a1e2a7789b, State: Initialized, Role: FOLLOWER
I20260929 10:58:45.532473 32120 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49227c63d1f4450084d265a1e2a7789b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49227c63d1f4450084d265a1e2a7789b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45315 } }
I20260929 10:58:45.532543 32120 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49227c63d1f4450084d265a1e2a7789b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:58:45.532570 32120 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49227c63d1f4450084d265a1e2a7789b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:58:45.532613 32120 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49227c63d1f4450084d265a1e2a7789b [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:58:45.533234 32120 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49227c63d1f4450084d265a1e2a7789b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49227c63d1f4450084d265a1e2a7789b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45315 } }
I20260929 10:58:45.533402 32120 leader_election.cc:302] T 00000000000000000000000000000000 P 49227c63d1f4450084d265a1e2a7789b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49227c63d1f4450084d265a1e2a7789b; no voters:
I20260929 10:58:45.533581 32120 leader_election.cc:288] T 00000000000000000000000000000000 P 49227c63d1f4450084d265a1e2a7789b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:58:45.533623 32125 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49227c63d1f4450084d265a1e2a7789b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:58:45.533812 32125 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49227c63d1f4450084d265a1e2a7789b [term 1 LEADER]: Becoming Leader. State: Replica: 49227c63d1f4450084d265a1e2a7789b, State: Running, Role: LEADER
I20260929 10:58:45.533878 32120 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49227c63d1f4450084d265a1e2a7789b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:58:45.533905 32125 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49227c63d1f4450084d265a1e2a7789b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49227c63d1f4450084d265a1e2a7789b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45315 } }
I20260929 10:58:45.534230 32125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49227c63d1f4450084d265a1e2a7789b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49227c63d1f4450084d265a1e2a7789b. Latest consensus state: current_term: 1 leader_uuid: "49227c63d1f4450084d265a1e2a7789b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49227c63d1f4450084d265a1e2a7789b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45315 } } }
I20260929 10:58:45.534332 32125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49227c63d1f4450084d265a1e2a7789b [sys.catalog]: This master's current role is: LEADER
I20260929 10:58:45.534602 32126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49227c63d1f4450084d265a1e2a7789b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49227c63d1f4450084d265a1e2a7789b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49227c63d1f4450084d265a1e2a7789b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45315 } } }
I20260929 10:58:45.534665 32126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49227c63d1f4450084d265a1e2a7789b [sys.catalog]: This master's current role is: LEADER
I20260929 10:58:45.534891 32129 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:58:45.535429 32129 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 29 10:58:45 dist-test-slave-8nrt krb5kdc[32039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 10:58:45.537248 32129 catalog_manager.cc:1347] Generated new cluster ID: 960b7e826c9a4d54919f73f4958ae3d9
I20260929 10:58:45.537314 32129 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:58:45.541759 32129 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:58:45.542447 32129 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:58:45.544183 32123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:45.529165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.42:43235 (local address 127.0.0.1:45315)
0929 10:58:45.529364 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:45.529373 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:45.529406 (+ 33us) server_negotiation.cc:407] Connection header received
0929 10:58:45.529924 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:45.529933 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:45.530000 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:45.530115 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:45.530930 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:45.531789 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:45.532895 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:45.533357 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:45.533485 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:45.536384 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:45.536388 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:45.536388 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:58:45.536400 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:45.542421 (+ 6021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:45.542997 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:45.543001 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:45.543002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:45.543111 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:45.543382 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:45.543383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:45.543383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:45.543579 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:45.543780 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:45.543867 (+ 87us) server_negotiation.cc:299] Negotiation successful
0929 10:58:45.543965 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":107,"threads_started":1}
I20260929 10:58:45.545179 11195 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:58:45.549536 32129 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49227c63d1f4450084d265a1e2a7789b: Generated new TSK 0
I20260929 10:58:45.549708 32129 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:58:45.555367 32123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:45.546585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.42:43303 (local address 127.0.0.1:45315)
0929 10:58:45.546622 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:45.546624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:45.549338 (+ 2714us) server_negotiation.cc:407] Connection header received
0929 10:58:45.549399 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:45.549399 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:45.549411 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:45.549473 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:45.550166 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:45.550686 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:45.551421 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:45.551634 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:45.551710 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:45.552885 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:45.552886 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:45.552887 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:58:45.552894 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:45.554221 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:45.554582 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:45.554583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:45.554583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:45.554644 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:45.554869 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:45.554870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:45.554871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:45.554979 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:45.555099 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:45.555210 (+ 111us) server_negotiation.cc:299] Negotiation successful
0929 10:58:45.555274 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 10:58:45.557214 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 32052
W20260929 10:58:45.561146 32037 connection.cc:535] client connection to 127.0.0.1:45315 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45315): Connection reset by peer (error 104)
2026-09-29T10:58:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1301 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790679274510869-11195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:58:45 dist-test-slave-8nrt krb5kdc[32151](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:58:45 dist-test-slave-8nrt krb5kdc[32151](info): set up 2 sockets
Sep 29 10:58:45 dist-test-slave-8nrt krb5kdc[32151](info): commencing operation
krb5kdc: starting...
W20260929 10:58:46.589392 11195 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Sep 29 10:58:46 dist-test-slave-8nrt krb5kdc[32151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:58:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:58:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790679274510869-11195-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790679274510869-11195-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 10:58:46.705896 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42421
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790679274510869-11195-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42421
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:58:46.789616 32164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:58:46.789999 32164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:58:46.790043 32164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:58:46.792232 32164 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:58:46.792313 32164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:58:46.792340 32164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:58:46.792397 32164 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:58:46.792429 32164 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:58:46.794620 32164 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790679274510869-11195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790679274510869-11195-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:58:46.795115 32164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:58:46.795432 32164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:58:46.800484 32172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:46.800457 32170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:46.800467 32169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:46.800817 32164 server_base.cc:1034] running on GCE node
I20260929 10:58:46.801055 32164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:58:46.801398 32164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:58:46.802564 32164 hybrid_clock.cc:648] HybridClock initialized: now 1790679526802542 us; error 40 us; skew 500 ppm
Sep 29 10:58:46 dist-test-slave-8nrt krb5kdc[32151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:58:46.804987 32164 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 10:58:46.805433 32164 webserver.cc:460] Webserver started at http://127.0.0.1:46115/ using document root <none> and password file <none>
I20260929 10:58:46.805626 32164 fs_manager.cc:359] Metadata directory not provided
I20260929 10:58:46.805666 32164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:58:46.805771 32164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:58:46.806758 32164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790679274510869-11195-0/minicluster-data/master-0/data/instance:
uuid: "e1c627dd65c74761b7bb7f1c9f8b8ada"
format_stamp: "Formatted at 2026-09-29 10:58:46 on dist-test-slave-8nrt"
server_key: "929e8f37e4a8715ed4b9ce33505f542c"
server_key_iv: "4ab466091d6e985f838e16eb5d536fbd"
server_key_version: "encryptionkey@0"
I20260929 10:58:46.807072 32164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790679274510869-11195-0/minicluster-data/master-0/wal/instance:
uuid: "e1c627dd65c74761b7bb7f1c9f8b8ada"
format_stamp: "Formatted at 2026-09-29 10:58:46 on dist-test-slave-8nrt"
server_key: "929e8f37e4a8715ed4b9ce33505f542c"
server_key_iv: "4ab466091d6e985f838e16eb5d536fbd"
server_key_version: "encryptionkey@0"
I20260929 10:58:46.808308 32164 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 10:58:46.808998 32179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:46.809152 32164 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 10:58:46.809247 32164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "e1c627dd65c74761b7bb7f1c9f8b8ada"
format_stamp: "Formatted at 2026-09-29 10:58:46 on dist-test-slave-8nrt"
server_key: "929e8f37e4a8715ed4b9ce33505f542c"
server_key_iv: "4ab466091d6e985f838e16eb5d536fbd"
server_key_version: "encryptionkey@0"
I20260929 10:58:46.809365 32164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:58:46.817034 32164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:58:46.819214 32164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:58:46.819329 32164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:58:46.823062 32164 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42421
I20260929 10:58:46.823137 32231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42421 every 8 connection(s)
I20260929 10:58:46.823463 32164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:58:46.824303 32232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:58:46.826581 32232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1c627dd65c74761b7bb7f1c9f8b8ada: Bootstrap starting.
I20260929 10:58:46.827143 32232 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1c627dd65c74761b7bb7f1c9f8b8ada: Neither blocks nor log segments found. Creating new log.
I20260929 10:58:46.827450 32232 log.cc:824] T 00000000000000000000000000000000 P e1c627dd65c74761b7bb7f1c9f8b8ada: Log is configured to *not* fsync() on all Append() calls
I20260929 10:58:46.828115 32232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1c627dd65c74761b7bb7f1c9f8b8ada: No bootstrap required, opened a new log
I20260929 10:58:46.829767 32232 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1c627dd65c74761b7bb7f1c9f8b8ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1c627dd65c74761b7bb7f1c9f8b8ada" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42421 } }
I20260929 10:58:46.829924 32232 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1c627dd65c74761b7bb7f1c9f8b8ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:58:46.829993 32232 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1c627dd65c74761b7bb7f1c9f8b8ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1c627dd65c74761b7bb7f1c9f8b8ada, State: Initialized, Role: FOLLOWER
I20260929 10:58:46.830191 32232 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1c627dd65c74761b7bb7f1c9f8b8ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1c627dd65c74761b7bb7f1c9f8b8ada" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42421 } }
I20260929 10:58:46.830292 32232 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1c627dd65c74761b7bb7f1c9f8b8ada [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:58:46.830338 32232 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1c627dd65c74761b7bb7f1c9f8b8ada [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:58:46.830387 32232 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1c627dd65c74761b7bb7f1c9f8b8ada [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:58:46.830788 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 32164
I20260929 10:58:46.830916 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790679274510869-11195-0/minicluster-data/master-0/wal/instance
I20260929 10:58:46.831079 32232 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1c627dd65c74761b7bb7f1c9f8b8ada [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1c627dd65c74761b7bb7f1c9f8b8ada" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42421 } }
I20260929 10:58:46.831149 11195 external_mini_cluster.cc:1428] Setting key b8b4a51dce825b74fe93e4197a757e06
I20260929 10:58:46.831199 32232 leader_election.cc:302] T 00000000000000000000000000000000 P e1c627dd65c74761b7bb7f1c9f8b8ada [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1c627dd65c74761b7bb7f1c9f8b8ada; no voters:
I20260929 10:58:46.831363 32232 leader_election.cc:288] T 00000000000000000000000000000000 P e1c627dd65c74761b7bb7f1c9f8b8ada [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:58:46.831403 32235 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1c627dd65c74761b7bb7f1c9f8b8ada [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:58:46.831564 32235 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1c627dd65c74761b7bb7f1c9f8b8ada [term 1 LEADER]: Becoming Leader. State: Replica: e1c627dd65c74761b7bb7f1c9f8b8ada, State: Running, Role: LEADER
I20260929 10:58:46.831717 32235 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1c627dd65c74761b7bb7f1c9f8b8ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1c627dd65c74761b7bb7f1c9f8b8ada" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42421 } }
I20260929 10:58:46.831598 32232 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1c627dd65c74761b7bb7f1c9f8b8ada [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:58:46.832101 32236 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1c627dd65c74761b7bb7f1c9f8b8ada [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1c627dd65c74761b7bb7f1c9f8b8ada" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1c627dd65c74761b7bb7f1c9f8b8ada" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42421 } } }
I20260929 10:58:46.832181 32236 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1c627dd65c74761b7bb7f1c9f8b8ada [sys.catalog]: This master's current role is: LEADER
I20260929 10:58:46.832350 32237 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1c627dd65c74761b7bb7f1c9f8b8ada [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1c627dd65c74761b7bb7f1c9f8b8ada. Latest consensus state: current_term: 1 leader_uuid: "e1c627dd65c74761b7bb7f1c9f8b8ada" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1c627dd65c74761b7bb7f1c9f8b8ada" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42421 } } }
I20260929 10:58:46.832403 32237 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1c627dd65c74761b7bb7f1c9f8b8ada [sys.catalog]: This master's current role is: LEADER
I20260929 10:58:46.832806 32241 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:58:46.833339 32241 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:58:46.834878 32241 catalog_manager.cc:1347] Generated new cluster ID: e00d7e3da6a7484c8a78570e67921696
I20260929 10:58:46.834934 32241 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:58:46.837954 32239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:46.831773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.42:36895 (local address 127.0.0.1:42421)
0929 10:58:46.832160 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:46.832169 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:46.832379 (+ 210us) server_negotiation.cc:407] Connection header received
0929 10:58:46.832618 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:46.832635 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:46.832709 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:46.833141 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:46.833245 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:46.834578 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:46.835706 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:46.836345 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:46.836468 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:46.837170 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:46.837174 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:46.837174 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:58:46.837468 (+ 294us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:46.837506 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:58:46.837523 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:46.837628 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:46.837713 (+ 85us) server_negotiation.cc:299] Negotiation successful
0929 10:58:46.837745 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":61,"threads_started":1}
I20260929 10:58:46.838868 11195 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:58:46.843802 32241 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:58:46.844483 32241 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:58:46.844909 32239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:46.840598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.42:50995 (local address 127.0.0.1:42421)
0929 10:58:46.840653 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:46.840655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:46.841439 (+ 784us) server_negotiation.cc:407] Connection header received
0929 10:58:46.841484 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:46.841485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:46.841496 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:46.841573 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:46.841674 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:46.842422 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:46.843617 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:46.843836 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:46.843902 (+ 66us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:46.844674 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:46.844675 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:46.844676 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:58:46.844680 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:46.844695 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:58:46.844699 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:46.844727 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:46.844834 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 10:58:46.844844 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 10:58:46.850764 32241 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1c627dd65c74761b7bb7f1c9f8b8ada: Generated new TSK 0
I20260929 10:58:46.850934 32241 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:58:46.854967 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 32164
W20260929 10:58:46.859704 32149 connection.cc:535] client connection to 127.0.0.1:42421 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42421): Connection reset by peer (error 104)
2026-09-29T10:58:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1298 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790679274510869-11195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:58:46 dist-test-slave-8nrt krb5kdc[32263](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:58:46 dist-test-slave-8nrt krb5kdc[32263](info): set up 2 sockets
Sep 29 10:58:46 dist-test-slave-8nrt krb5kdc[32263](info): commencing operation
krb5kdc: starting...
W20260929 10:58:47.969353 11195 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Sep 29 10:58:48 dist-test-slave-8nrt krb5kdc[32263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:58:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:58:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790679274510869-11195-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790679274510869-11195-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 10:58:48.092051 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36735
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790679274510869-11195-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36735
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:58:48.178998 32276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:58:48.179211 32276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:58:48.179236 32276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:58:48.180342 32276 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:58:48.180384 32276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:58:48.180401 32276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:58:48.180414 32276 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:58:48.180426 32276 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:58:48.181874 32276 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790679274510869-11195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36735
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790679274510869-11195-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:58:48.182072 32276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:58:48.182250 32276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:58:48.187070 32281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:48.187074 32284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:48.187077 32282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:48.187386 32276 server_base.cc:1034] running on GCE node
I20260929 10:58:48.187661 32276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:58:48.187989 32276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:58:48.189215 32276 hybrid_clock.cc:648] HybridClock initialized: now 1790679528189200 us; error 47 us; skew 500 ppm
Sep 29 10:58:48 dist-test-slave-8nrt krb5kdc[32263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:58:48.191989 32276 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 10:58:48.192456 32276 webserver.cc:460] Webserver started at http://127.0.0.1:39997/ using document root <none> and password file <none>
I20260929 10:58:48.192726 32276 fs_manager.cc:359] Metadata directory not provided
I20260929 10:58:48.192803 32276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:58:48.192967 32276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:58:48.194242 32276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790679274510869-11195-0/minicluster-data/master-0/data/instance:
uuid: "33f7faf95c53484385cb30a95e53ad15"
format_stamp: "Formatted at 2026-09-29 10:58:48 on dist-test-slave-8nrt"
server_key: "74c24d7073931ac962ccb3fdb59c2084"
server_key_iv: "f4835cb81abafff06c404e3e039f820d"
server_key_version: "encryptionkey@0"
I20260929 10:58:48.194643 32276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790679274510869-11195-0/minicluster-data/master-0/wal/instance:
uuid: "33f7faf95c53484385cb30a95e53ad15"
format_stamp: "Formatted at 2026-09-29 10:58:48 on dist-test-slave-8nrt"
server_key: "74c24d7073931ac962ccb3fdb59c2084"
server_key_iv: "f4835cb81abafff06c404e3e039f820d"
server_key_version: "encryptionkey@0"
I20260929 10:58:48.196158 32276 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 10:58:48.197022 32291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:48.197235 32276 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:58:48.197396 32276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "33f7faf95c53484385cb30a95e53ad15"
format_stamp: "Formatted at 2026-09-29 10:58:48 on dist-test-slave-8nrt"
server_key: "74c24d7073931ac962ccb3fdb59c2084"
server_key_iv: "f4835cb81abafff06c404e3e039f820d"
server_key_version: "encryptionkey@0"
I20260929 10:58:48.197481 32276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:58:48.200954 32276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:58:48.202870 32276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:58:48.203024 32276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:58:48.206939 32276 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36735
I20260929 10:58:48.206977 32343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36735 every 8 connection(s)
I20260929 10:58:48.207321 32276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:58:48.208316 32344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:58:48.210764 32344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33f7faf95c53484385cb30a95e53ad15: Bootstrap starting.
I20260929 10:58:48.211385 32344 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33f7faf95c53484385cb30a95e53ad15: Neither blocks nor log segments found. Creating new log.
I20260929 10:58:48.211681 32344 log.cc:824] T 00000000000000000000000000000000 P 33f7faf95c53484385cb30a95e53ad15: Log is configured to *not* fsync() on all Append() calls
I20260929 10:58:48.212360 32344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33f7faf95c53484385cb30a95e53ad15: No bootstrap required, opened a new log
I20260929 10:58:48.214066 32344 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33f7faf95c53484385cb30a95e53ad15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33f7faf95c53484385cb30a95e53ad15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36735 } }
I20260929 10:58:48.214200 32344 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33f7faf95c53484385cb30a95e53ad15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:58:48.214222 32344 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33f7faf95c53484385cb30a95e53ad15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33f7faf95c53484385cb30a95e53ad15, State: Initialized, Role: FOLLOWER
I20260929 10:58:48.214391 32344 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33f7faf95c53484385cb30a95e53ad15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33f7faf95c53484385cb30a95e53ad15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36735 } }
I20260929 10:58:48.214468 32344 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33f7faf95c53484385cb30a95e53ad15 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:58:48.214510 32344 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33f7faf95c53484385cb30a95e53ad15 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:58:48.214560 32344 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33f7faf95c53484385cb30a95e53ad15 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:58:48.215080 32344 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33f7faf95c53484385cb30a95e53ad15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33f7faf95c53484385cb30a95e53ad15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36735 } }
I20260929 10:58:48.215189 32344 leader_election.cc:302] T 00000000000000000000000000000000 P 33f7faf95c53484385cb30a95e53ad15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33f7faf95c53484385cb30a95e53ad15; no voters:
I20260929 10:58:48.215369 32344 leader_election.cc:288] T 00000000000000000000000000000000 P 33f7faf95c53484385cb30a95e53ad15 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:58:48.215404 32347 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33f7faf95c53484385cb30a95e53ad15 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:58:48.215571 32347 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33f7faf95c53484385cb30a95e53ad15 [term 1 LEADER]: Becoming Leader. State: Replica: 33f7faf95c53484385cb30a95e53ad15, State: Running, Role: LEADER
I20260929 10:58:48.215621 32344 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33f7faf95c53484385cb30a95e53ad15 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:58:48.215687 32347 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33f7faf95c53484385cb30a95e53ad15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33f7faf95c53484385cb30a95e53ad15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36735 } }
I20260929 10:58:48.216061 32349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33f7faf95c53484385cb30a95e53ad15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33f7faf95c53484385cb30a95e53ad15. Latest consensus state: current_term: 1 leader_uuid: "33f7faf95c53484385cb30a95e53ad15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33f7faf95c53484385cb30a95e53ad15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36735 } } }
I20260929 10:58:48.216188 32348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33f7faf95c53484385cb30a95e53ad15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33f7faf95c53484385cb30a95e53ad15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33f7faf95c53484385cb30a95e53ad15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36735 } } }
I20260929 10:58:48.216202 32349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33f7faf95c53484385cb30a95e53ad15 [sys.catalog]: This master's current role is: LEADER
I20260929 10:58:48.216291 32348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33f7faf95c53484385cb30a95e53ad15 [sys.catalog]: This master's current role is: LEADER
I20260929 10:58:48.216836 32356 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:58:48.217209 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 32276
I20260929 10:58:48.217381 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790679274510869-11195-0/minicluster-data/master-0/wal/instance
I20260929 10:58:48.217398 32356 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:58:48.217605 11195 external_mini_cluster.cc:1428] Setting key 5ee8675a59b930e348e699d79fb60aae
I20260929 10:58:48.218941 32356 catalog_manager.cc:1347] Generated new cluster ID: 2f830e8092e440f0a63c81f369583743
I20260929 10:58:48.219002 32356 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:58:48.220314 32366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:48.218090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.42:36655 (local address 127.0.0.1:36735)
0929 10:58:48.218240 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:48.218249 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:48.218435 (+ 186us) server_negotiation.cc:407] Connection header received
0929 10:58:48.218608 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:48.218616 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:48.218663 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:48.218789 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:48.219874 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:48.219878 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:48.219878 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:58:48.219882 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:48.219923 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:58:48.219938 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:48.220012 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:48.220085 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 10:58:48.220135 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":69,"threads_started":1}
I20260929 10:58:48.221194 11195 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:58:48.226712 32366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:48.225541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.42:34509 (local address 127.0.0.1:36735)
0929 10:58:48.225580 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:48.225582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:48.225768 (+ 186us) server_negotiation.cc:407] Connection header received
0929 10:58:48.225829 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:48.225830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:48.225835 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:48.225886 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:48.226492 (+ 606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:48.226492 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:48.226493 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:58:48.226497 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:48.226511 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:58:48.226515 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:48.226569 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:48.226637 (+ 68us) server_negotiation.cc:299] Negotiation successful
0929 10:58:48.226649 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 10:58:48.231907 32356 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:58:48.232088 32356 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:58:48.241194 32356 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33f7faf95c53484385cb30a95e53ad15: Generated new TSK 0
I20260929 10:58:48.241405 32356 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:58:48.248423 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 32276
2026-09-29T10:58:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1392 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790679274510869-11195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:58:48 dist-test-slave-8nrt krb5kdc[32375](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:58:48 dist-test-slave-8nrt krb5kdc[32375](info): set up 2 sockets
Sep 29 10:58:48 dist-test-slave-8nrt krb5kdc[32375](info): commencing operation
krb5kdc: starting...
W20260929 10:58:49.277494 11195 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Sep 29 10:58:49 dist-test-slave-8nrt krb5kdc[32375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:58:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:58:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790679274510869-11195-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790679274510869-11195-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 10:58:49.394219 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35061
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790679274510869-11195-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:58:49.476287 32388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:58:49.476548 32388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:58:49.476588 32388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:58:49.477804 32388 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:58:49.477869 32388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:58:49.477896 32388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:58:49.477928 32388 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:58:49.477950 32388 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:58:49.479387 32388 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790679274510869-11195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35061
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790679274510869-11195-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:58:49.479667 32388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:58:49.479893 32388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:58:49.484925 32393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:49.484933 32396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:49.485189 32388 server_base.cc:1034] running on GCE node
W20260929 10:58:49.484933 32394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:49.485526 32388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:58:49.485853 32388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:58:49.487044 32388 hybrid_clock.cc:648] HybridClock initialized: now 1790679529487039 us; error 36 us; skew 500 ppm
Sep 29 10:58:49 dist-test-slave-8nrt krb5kdc[32375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:58:49.489713 32388 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 10:58:49.490163 32388 webserver.cc:460] Webserver started at http://127.0.0.1:41427/ using document root <none> and password file <none>
I20260929 10:58:49.490355 32388 fs_manager.cc:359] Metadata directory not provided
I20260929 10:58:49.490393 32388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:58:49.490479 32388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:58:49.491499 32388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790679274510869-11195-0/minicluster-data/master-0/data/instance:
uuid: "5a409debb89345eb817e2fed007e479c"
format_stamp: "Formatted at 2026-09-29 10:58:49 on dist-test-slave-8nrt"
server_key: "0c2e573dca573a4c960edde5b5199b9e"
server_key_iv: "e009d7600ba49b5a4e2efa8ad8de5f2b"
server_key_version: "encryptionkey@0"
I20260929 10:58:49.491815 32388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790679274510869-11195-0/minicluster-data/master-0/wal/instance:
uuid: "5a409debb89345eb817e2fed007e479c"
format_stamp: "Formatted at 2026-09-29 10:58:49 on dist-test-slave-8nrt"
server_key: "0c2e573dca573a4c960edde5b5199b9e"
server_key_iv: "e009d7600ba49b5a4e2efa8ad8de5f2b"
server_key_version: "encryptionkey@0"
I20260929 10:58:49.493080 32388 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 10:58:49.493903 32403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:49.494097 32388 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 10:58:49.494189 32388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "5a409debb89345eb817e2fed007e479c"
format_stamp: "Formatted at 2026-09-29 10:58:49 on dist-test-slave-8nrt"
server_key: "0c2e573dca573a4c960edde5b5199b9e"
server_key_iv: "e009d7600ba49b5a4e2efa8ad8de5f2b"
server_key_version: "encryptionkey@0"
I20260929 10:58:49.494247 32388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:58:49.508569 32388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:58:49.510970 32388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:58:49.511130 32388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:58:49.514981 32388 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35061
I20260929 10:58:49.515035 32455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35061 every 8 connection(s)
I20260929 10:58:49.515358 32388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:58:49.516203 32456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:58:49.518473 32456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a409debb89345eb817e2fed007e479c: Bootstrap starting.
I20260929 10:58:49.518939 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 32388
I20260929 10:58:49.519078 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790679274510869-11195-0/minicluster-data/master-0/wal/instance
I20260929 10:58:49.519173 32456 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a409debb89345eb817e2fed007e479c: Neither blocks nor log segments found. Creating new log.
I20260929 10:58:49.519337 11195 external_mini_cluster.cc:1428] Setting key 26047d17e07d1066bc24f7cf9f33b1b4
I20260929 10:58:49.519531 32456 log.cc:824] T 00000000000000000000000000000000 P 5a409debb89345eb817e2fed007e479c: Log is configured to *not* fsync() on all Append() calls
I20260929 10:58:49.520329 32456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a409debb89345eb817e2fed007e479c: No bootstrap required, opened a new log
I20260929 10:58:49.522799 32456 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a409debb89345eb817e2fed007e479c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a409debb89345eb817e2fed007e479c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35061 } }
I20260929 10:58:49.522956 32456 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a409debb89345eb817e2fed007e479c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:58:49.522987 32456 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a409debb89345eb817e2fed007e479c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a409debb89345eb817e2fed007e479c, State: Initialized, Role: FOLLOWER
I20260929 10:58:49.523124 32456 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a409debb89345eb817e2fed007e479c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a409debb89345eb817e2fed007e479c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35061 } }
I20260929 10:58:49.523222 32456 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a409debb89345eb817e2fed007e479c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:58:49.523259 32456 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a409debb89345eb817e2fed007e479c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:58:49.523307 32456 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a409debb89345eb817e2fed007e479c [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:58:49.523856 32456 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a409debb89345eb817e2fed007e479c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a409debb89345eb817e2fed007e479c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35061 } }
I20260929 10:58:49.523960 32456 leader_election.cc:302] T 00000000000000000000000000000000 P 5a409debb89345eb817e2fed007e479c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a409debb89345eb817e2fed007e479c; no voters:
I20260929 10:58:49.524106 32456 leader_election.cc:288] T 00000000000000000000000000000000 P 5a409debb89345eb817e2fed007e479c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:58:49.524166 32461 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a409debb89345eb817e2fed007e479c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:58:49.524328 32456 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a409debb89345eb817e2fed007e479c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:58:49.524374 32461 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a409debb89345eb817e2fed007e479c [term 1 LEADER]: Becoming Leader. State: Replica: 5a409debb89345eb817e2fed007e479c, State: Running, Role: LEADER
I20260929 10:58:49.524514 32461 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a409debb89345eb817e2fed007e479c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a409debb89345eb817e2fed007e479c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35061 } }
I20260929 10:58:49.525204 32462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a409debb89345eb817e2fed007e479c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a409debb89345eb817e2fed007e479c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a409debb89345eb817e2fed007e479c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35061 } } }
I20260929 10:58:49.525308 32462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a409debb89345eb817e2fed007e479c [sys.catalog]: This master's current role is: LEADER
I20260929 10:58:49.525286 32463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a409debb89345eb817e2fed007e479c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a409debb89345eb817e2fed007e479c. Latest consensus state: current_term: 1 leader_uuid: "5a409debb89345eb817e2fed007e479c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a409debb89345eb817e2fed007e479c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35061 } } }
I20260929 10:58:49.525405 32463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a409debb89345eb817e2fed007e479c [sys.catalog]: This master's current role is: LEADER
Sep 29 10:58:49 dist-test-slave-8nrt krb5kdc[32375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
W20260929 10:58:49.527225 32476 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5a409debb89345eb817e2fed007e479c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:58:49.527455 32476 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:58:49.527545 32468 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:58:49.527755 32468 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:58:49.529318 32468 catalog_manager.cc:1347] Generated new cluster ID: fe979580e81c4479ba370c555cf778da
I20260929 10:58:49.529373 32468 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:58:49.530815 32460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:49.519885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.42:41889 (local address 127.0.0.1:35061)
0929 10:58:49.520021 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:49.520029 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:49.520062 (+ 33us) server_negotiation.cc:407] Connection header received
0929 10:58:49.520668 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:49.520676 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:49.520739 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:49.520883 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:49.521527 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:49.522990 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:49.523832 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:49.524457 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:49.524935 (+ 478us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:49.527141 (+ 2206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:49.527145 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:49.527145 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:58:49.527158 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:49.529138 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:49.529646 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:49.529649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:49.529649 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:49.529745 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:49.529986 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:49.529987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:49.529987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:49.530225 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:49.530409 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:49.530515 (+ 106us) server_negotiation.cc:299] Negotiation successful
0929 10:58:49.530617 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":60,"threads_started":1}
I20260929 10:58:49.531767 11195 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:58:49.539366 32460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:49.533147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.42:55593 (local address 127.0.0.1:35061)
0929 10:58:49.533183 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:49.533184 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:49.533343 (+ 159us) server_negotiation.cc:407] Connection header received
0929 10:58:49.533423 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:49.533423 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:49.533433 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:49.533476 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:49.534035 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:49.534519 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:49.535328 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:49.535563 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:49.535640 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:49.536786 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:49.536788 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:49.536789 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:58:49.536795 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:49.538200 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:49.538559 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:49.538560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:49.538560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:49.538612 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:49.538844 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:49.538846 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:49.538846 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:49.538955 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:49.539064 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:49.539196 (+ 132us) server_negotiation.cc:299] Negotiation successful
0929 10:58:49.539256 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 10:58:49.543582 32468 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:58:49.544065 32468 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:58:49.550673 32468 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a409debb89345eb817e2fed007e479c: Generated new TSK 0
I20260929 10:58:49.550871 32468 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:58:49.555938 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 32388
W20260929 10:58:49.560148 32373 connection.cc:535] client connection to 127.0.0.1:35061 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35061): Connection reset by peer (error 104)
2026-09-29T10:58:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1307 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790679274510869-11195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:58:49 dist-test-slave-8nrt krb5kdc[32487](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:58:49 dist-test-slave-8nrt krb5kdc[32487](info): set up 2 sockets
Sep 29 10:58:49 dist-test-slave-8nrt krb5kdc[32487](info): commencing operation
krb5kdc: starting...
W20260929 10:58:50.609395 11195 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Sep 29 10:58:50 dist-test-slave-8nrt krb5kdc[32487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:58:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:58:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790679274510869-11195-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790679274510869-11195-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 10:58:50.725951 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33801
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790679274510869-11195-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33801
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:58:50.814540 32500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:58:50.814872 32500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:58:50.814913 32500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:58:50.816720 32500 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:58:50.816820 32500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:58:50.816872 32500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:58:50.816906 32500 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:58:50.816933 32500 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:58:50.818815 32500 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790679274510869-11195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33801
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790679274510869-11195-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:58:50.819100 32500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:58:50.819352 32500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 10:58:50.824355 32500 server_base.cc:1034] running on GCE node
W20260929 10:58:50.824455 32508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:50.824373 32506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:50.824343 32505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:50.824872 32500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:58:50.825163 32500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:58:50.826301 32500 hybrid_clock.cc:648] HybridClock initialized: now 1790679530826301 us; error 38 us; skew 500 ppm
Sep 29 10:58:50 dist-test-slave-8nrt krb5kdc[32487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679530, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:58:50.828768 32500 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 10:58:50.829172 32500 webserver.cc:460] Webserver started at http://127.0.0.1:46193/ using document root <none> and password file <none>
I20260929 10:58:50.829432 32500 fs_manager.cc:359] Metadata directory not provided
I20260929 10:58:50.829502 32500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:58:50.829650 32500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:58:50.830691 32500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790679274510869-11195-0/minicluster-data/master-0/data/instance:
uuid: "fc2e0e9270ff46e89ee6fee226ee47b2"
format_stamp: "Formatted at 2026-09-29 10:58:50 on dist-test-slave-8nrt"
server_key: "eb3055b1ed2f302ba85c296bbf65ec56"
server_key_iv: "5b7a22a46774648ed2741ce3ef55fa07"
server_key_version: "encryptionkey@0"
I20260929 10:58:50.831053 32500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790679274510869-11195-0/minicluster-data/master-0/wal/instance:
uuid: "fc2e0e9270ff46e89ee6fee226ee47b2"
format_stamp: "Formatted at 2026-09-29 10:58:50 on dist-test-slave-8nrt"
server_key: "eb3055b1ed2f302ba85c296bbf65ec56"
server_key_iv: "5b7a22a46774648ed2741ce3ef55fa07"
server_key_version: "encryptionkey@0"
I20260929 10:58:50.832370 32500 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 10:58:50.833073 32515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:50.833318 32500 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:58:50.833442 32500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "fc2e0e9270ff46e89ee6fee226ee47b2"
format_stamp: "Formatted at 2026-09-29 10:58:50 on dist-test-slave-8nrt"
server_key: "eb3055b1ed2f302ba85c296bbf65ec56"
server_key_iv: "5b7a22a46774648ed2741ce3ef55fa07"
server_key_version: "encryptionkey@0"
I20260929 10:58:50.833526 32500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:58:50.845358 32500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:58:50.847738 32500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:58:50.847901 32500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:58:50.851818 32500 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33801
I20260929 10:58:50.851856 32567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33801 every 8 connection(s)
I20260929 10:58:50.852228 32500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:58:50.853144 32568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:58:50.855515 32568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc2e0e9270ff46e89ee6fee226ee47b2: Bootstrap starting.
I20260929 10:58:50.856115 32568 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc2e0e9270ff46e89ee6fee226ee47b2: Neither blocks nor log segments found. Creating new log.
I20260929 10:58:50.856419 32568 log.cc:824] T 00000000000000000000000000000000 P fc2e0e9270ff46e89ee6fee226ee47b2: Log is configured to *not* fsync() on all Append() calls
I20260929 10:58:50.857137 32568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc2e0e9270ff46e89ee6fee226ee47b2: No bootstrap required, opened a new log
I20260929 10:58:50.858783 32568 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc2e0e9270ff46e89ee6fee226ee47b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc2e0e9270ff46e89ee6fee226ee47b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33801 } }
I20260929 10:58:50.858922 32568 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc2e0e9270ff46e89ee6fee226ee47b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:58:50.858945 32568 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc2e0e9270ff46e89ee6fee226ee47b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc2e0e9270ff46e89ee6fee226ee47b2, State: Initialized, Role: FOLLOWER
I20260929 10:58:50.859143 32568 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc2e0e9270ff46e89ee6fee226ee47b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc2e0e9270ff46e89ee6fee226ee47b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33801 } }
I20260929 10:58:50.859238 32568 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc2e0e9270ff46e89ee6fee226ee47b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:58:50.859280 32568 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc2e0e9270ff46e89ee6fee226ee47b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:58:50.859332 32568 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc2e0e9270ff46e89ee6fee226ee47b2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:58:50.859866 32568 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc2e0e9270ff46e89ee6fee226ee47b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc2e0e9270ff46e89ee6fee226ee47b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33801 } }
I20260929 10:58:50.859979 32568 leader_election.cc:302] T 00000000000000000000000000000000 P fc2e0e9270ff46e89ee6fee226ee47b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc2e0e9270ff46e89ee6fee226ee47b2; no voters:
I20260929 10:58:50.860169 32568 leader_election.cc:288] T 00000000000000000000000000000000 P fc2e0e9270ff46e89ee6fee226ee47b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:58:50.860234 32571 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc2e0e9270ff46e89ee6fee226ee47b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:58:50.860419 32571 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc2e0e9270ff46e89ee6fee226ee47b2 [term 1 LEADER]: Becoming Leader. State: Replica: fc2e0e9270ff46e89ee6fee226ee47b2, State: Running, Role: LEADER
I20260929 10:58:50.860419 32568 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc2e0e9270ff46e89ee6fee226ee47b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:58:50.860589 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 32500
I20260929 10:58:50.860574 32571 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc2e0e9270ff46e89ee6fee226ee47b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc2e0e9270ff46e89ee6fee226ee47b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33801 } }
I20260929 10:58:50.860715 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790679274510869-11195-0/minicluster-data/master-0/wal/instance
I20260929 10:58:50.860945 11195 external_mini_cluster.cc:1428] Setting key c11a7f9bc7051a0182760341954fc67c
I20260929 10:58:50.861111 32573 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc2e0e9270ff46e89ee6fee226ee47b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc2e0e9270ff46e89ee6fee226ee47b2. Latest consensus state: current_term: 1 leader_uuid: "fc2e0e9270ff46e89ee6fee226ee47b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc2e0e9270ff46e89ee6fee226ee47b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33801 } } }
I20260929 10:58:50.861129 32572 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc2e0e9270ff46e89ee6fee226ee47b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc2e0e9270ff46e89ee6fee226ee47b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc2e0e9270ff46e89ee6fee226ee47b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33801 } } }
I20260929 10:58:50.861214 32573 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc2e0e9270ff46e89ee6fee226ee47b2 [sys.catalog]: This master's current role is: LEADER
I20260929 10:58:50.861248 32572 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc2e0e9270ff46e89ee6fee226ee47b2 [sys.catalog]: This master's current role is: LEADER
I20260929 10:58:50.861488 32580 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:58:50.862056 32580 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:58:50.863698 32580 catalog_manager.cc:1347] Generated new cluster ID: 8be0751be97b472f98df5cf17766ca3a
I20260929 10:58:50.863749 32580 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:58:50.867558 32583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:50.861634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.42:55417 (local address 127.0.0.1:33801)
0929 10:58:50.862160 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:50.862171 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:50.862202 (+ 31us) server_negotiation.cc:407] Connection header received
0929 10:58:50.862387 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:50.862396 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:50.862458 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:50.862595 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:50.862859 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:50.864188 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:50.865316 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:50.865950 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:50.866059 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:50.866782 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:50.866786 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:50.866786 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:58:50.866988 (+ 202us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:50.867026 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:58:50.867041 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:50.867128 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:50.867314 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 10:58:50.867346 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":112,"threads_started":1}
I20260929 10:58:50.868570 11195 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:58:50.873409 32583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:50.870080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.42:46477 (local address 127.0.0.1:33801)
0929 10:58:50.870110 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:50.870112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:50.870120 (+ 8us) server_negotiation.cc:407] Connection header received
0929 10:58:50.870164 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:50.870164 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:50.870174 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:50.870239 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:50.870347 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:50.871121 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:50.872080 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:50.872394 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:50.872466 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:50.873100 (+ 634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:50.873101 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:50.873102 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:58:50.873106 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:50.873138 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:58:50.873143 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:50.873211 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:50.873325 (+ 114us) server_negotiation.cc:299] Negotiation successful
0929 10:58:50.873338 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260929 10:58:50.882283 32580 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:58:50.882790 32580 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:58:50.886242 32580 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc2e0e9270ff46e89ee6fee226ee47b2: Generated new TSK 0
I20260929 10:58:50.886400 32580 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:58:50.889034 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 32500
W20260929 10:58:50.892887 32485 connection.cc:535] client connection to 127.0.0.1:33801 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33801): Connection reset by peer (error 104)
2026-09-29T10:58:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1332 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790679274510869-11195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:58:50 dist-test-slave-8nrt krb5kdc[32599](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:58:50 dist-test-slave-8nrt krb5kdc[32599](info): set up 2 sockets
Sep 29 10:58:50 dist-test-slave-8nrt krb5kdc[32599](info): commencing operation
krb5kdc: starting...
W20260929 10:58:51.977398 11195 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Sep 29 10:58:52 dist-test-slave-8nrt krb5kdc[32599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:58:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:58:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790679274510869-11195-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790679274510869-11195-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 10:58:52.096273 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36315
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790679274510869-11195-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:58:52.176249 32612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:58:52.176519 32612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:58:52.176561 32612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:58:52.177757 32612 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:58:52.177829 32612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:58:52.177853 32612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:58:52.177875 32612 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:58:52.177906 32612 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:58:52.179651 32612 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790679274510869-11195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36315
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790679274510869-11195-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:58:52.179960 32612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:58:52.180213 32612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:58:52.185315 32618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:52.185331 32617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:52.185334 32620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:52.185526 32612 server_base.cc:1034] running on GCE node
I20260929 10:58:52.185813 32612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:58:52.186128 32612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:58:52.187287 32612 hybrid_clock.cc:648] HybridClock initialized: now 1790679532187269 us; error 40 us; skew 500 ppm
Sep 29 10:58:52 dist-test-slave-8nrt krb5kdc[32599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:58:52.189745 32612 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 10:58:52.190192 32612 webserver.cc:460] Webserver started at http://127.0.0.1:45759/ using document root <none> and password file <none>
I20260929 10:58:52.190372 32612 fs_manager.cc:359] Metadata directory not provided
I20260929 10:58:52.190412 32612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:58:52.190505 32612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:58:52.191576 32612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790679274510869-11195-0/minicluster-data/master-0/data/instance:
uuid: "16c82c8de6dc43d098dccd1090fe85b6"
format_stamp: "Formatted at 2026-09-29 10:58:52 on dist-test-slave-8nrt"
server_key: "153351bbaf55b8d75c19c1aa3c939f15"
server_key_iv: "9481eb6fe8a7960efb1052154cd59c09"
server_key_version: "encryptionkey@0"
I20260929 10:58:52.191881 32612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790679274510869-11195-0/minicluster-data/master-0/wal/instance:
uuid: "16c82c8de6dc43d098dccd1090fe85b6"
format_stamp: "Formatted at 2026-09-29 10:58:52 on dist-test-slave-8nrt"
server_key: "153351bbaf55b8d75c19c1aa3c939f15"
server_key_iv: "9481eb6fe8a7960efb1052154cd59c09"
server_key_version: "encryptionkey@0"
I20260929 10:58:52.193142 32612 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 10:58:52.193970 32627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:52.194177 32612 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:58:52.194252 32612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "16c82c8de6dc43d098dccd1090fe85b6"
format_stamp: "Formatted at 2026-09-29 10:58:52 on dist-test-slave-8nrt"
server_key: "153351bbaf55b8d75c19c1aa3c939f15"
server_key_iv: "9481eb6fe8a7960efb1052154cd59c09"
server_key_version: "encryptionkey@0"
I20260929 10:58:52.194343 32612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:58:52.197710 32612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:58:52.199553 32612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:58:52.199708 32612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:58:52.203795 32612 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36315
I20260929 10:58:52.203825 32679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36315 every 8 connection(s)
I20260929 10:58:52.204149 32612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:58:52.205119 32680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:58:52.207392 32680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16c82c8de6dc43d098dccd1090fe85b6: Bootstrap starting.
I20260929 10:58:52.208117 32680 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16c82c8de6dc43d098dccd1090fe85b6: Neither blocks nor log segments found. Creating new log.
I20260929 10:58:52.208448 32680 log.cc:824] T 00000000000000000000000000000000 P 16c82c8de6dc43d098dccd1090fe85b6: Log is configured to *not* fsync() on all Append() calls
I20260929 10:58:52.209225 32680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16c82c8de6dc43d098dccd1090fe85b6: No bootstrap required, opened a new log
I20260929 10:58:52.210845 32680 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16c82c8de6dc43d098dccd1090fe85b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16c82c8de6dc43d098dccd1090fe85b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36315 } }
I20260929 10:58:52.210983 32680 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16c82c8de6dc43d098dccd1090fe85b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:58:52.211005 32680 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16c82c8de6dc43d098dccd1090fe85b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16c82c8de6dc43d098dccd1090fe85b6, State: Initialized, Role: FOLLOWER
I20260929 10:58:52.211048 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 32612
I20260929 10:58:52.211185 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790679274510869-11195-0/minicluster-data/master-0/wal/instance
I20260929 10:58:52.211170 32680 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16c82c8de6dc43d098dccd1090fe85b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16c82c8de6dc43d098dccd1090fe85b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36315 } }
I20260929 10:58:52.211256 32680 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16c82c8de6dc43d098dccd1090fe85b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:58:52.211297 32680 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16c82c8de6dc43d098dccd1090fe85b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:58:52.211340 32680 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16c82c8de6dc43d098dccd1090fe85b6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:58:52.211432 11195 external_mini_cluster.cc:1428] Setting key 3f197b91857f92fd7633eb8016b9b53f
I20260929 10:58:52.212041 32680 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16c82c8de6dc43d098dccd1090fe85b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16c82c8de6dc43d098dccd1090fe85b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36315 } }
I20260929 10:58:52.212162 32680 leader_election.cc:302] T 00000000000000000000000000000000 P 16c82c8de6dc43d098dccd1090fe85b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16c82c8de6dc43d098dccd1090fe85b6; no voters:
I20260929 10:58:52.212352 32680 leader_election.cc:288] T 00000000000000000000000000000000 P 16c82c8de6dc43d098dccd1090fe85b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:58:52.212400 32685 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16c82c8de6dc43d098dccd1090fe85b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:58:52.212636 32685 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16c82c8de6dc43d098dccd1090fe85b6 [term 1 LEADER]: Becoming Leader. State: Replica: 16c82c8de6dc43d098dccd1090fe85b6, State: Running, Role: LEADER
I20260929 10:58:52.212710 32680 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16c82c8de6dc43d098dccd1090fe85b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:58:52.212792 32685 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16c82c8de6dc43d098dccd1090fe85b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16c82c8de6dc43d098dccd1090fe85b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36315 } }
I20260929 10:58:52.213707 32684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:52.211910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.42:37619 (local address 127.0.0.1:36315)
0929 10:58:52.212060 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:52.212068 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:52.212094 (+ 26us) server_negotiation.cc:407] Connection header received
0929 10:58:52.212242 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:52.212249 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:52.212280 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:52.212441 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:52.213188 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:52.213192 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:52.213192 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:58:52.213197 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:52.213249 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:58:52.213299 (+ 50us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:52.213379 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:52.213452 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 10:58:52.213501 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":69,"threads_started":1}
I20260929 10:58:52.213718 32685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16c82c8de6dc43d098dccd1090fe85b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16c82c8de6dc43d098dccd1090fe85b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16c82c8de6dc43d098dccd1090fe85b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36315 } } }
I20260929 10:58:52.213829 32685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16c82c8de6dc43d098dccd1090fe85b6 [sys.catalog]: This master's current role is: LEADER
I20260929 10:58:52.213956 32685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16c82c8de6dc43d098dccd1090fe85b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16c82c8de6dc43d098dccd1090fe85b6. Latest consensus state: current_term: 1 leader_uuid: "16c82c8de6dc43d098dccd1090fe85b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16c82c8de6dc43d098dccd1090fe85b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36315 } } }
I20260929 10:58:52.214001 32685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16c82c8de6dc43d098dccd1090fe85b6 [sys.catalog]: This master's current role is: LEADER
I20260929 10:58:52.214013 32697 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:58:52.214578 32697 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:58:52.214843 11195 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:58:52.216147 32697 catalog_manager.cc:1347] Generated new cluster ID: 45eb1387542949b99b946820579a1bed
I20260929 10:58:52.216200 32697 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:58:52.217509 32684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:52.216370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.42:53367 (local address 127.0.0.1:36315)
0929 10:58:52.216409 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:52.216411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:52.216508 (+ 97us) server_negotiation.cc:407] Connection header received
0929 10:58:52.216586 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:52.216587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:52.216592 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:52.216647 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:52.217282 (+ 635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:52.217283 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:52.217283 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:58:52.217287 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:52.217300 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:58:52.217304 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:52.217353 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:52.217415 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 10:58:52.217429 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 10:58:52.226218 32697 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:58:52.226402 32697 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:58:52.228403 32697 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16c82c8de6dc43d098dccd1090fe85b6: Generated new TSK 0
I20260929 10:58:52.228569 32697 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:58:52.229977 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 32612
2026-09-29T10:58:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1341 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790679274510869-11195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:58:52 dist-test-slave-8nrt krb5kdc[32711](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:58:52 dist-test-slave-8nrt krb5kdc[32711](info): set up 2 sockets
Sep 29 10:58:52 dist-test-slave-8nrt krb5kdc[32711](info): commencing operation
krb5kdc: starting...
W20260929 10:58:53.269614 11195 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Sep 29 10:58:53 dist-test-slave-8nrt krb5kdc[32711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:58:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:58:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
I20260929 10:58:53.392817 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:36957
--webserver_interface=127.10.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:41417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:36957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:58:53.472339 32724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:58:53.472563 32724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:58:53.472599 32724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:58:53.473722 32724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:58:53.473793 32724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:58:53.473819 32724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:58:53.473852 32724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:58:53.473877 32724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:58:53.475301 32724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:41417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790679274510869-11195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:36957
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:36957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:58:53.475564 32724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:58:53.475800 32724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:58:53.480589 32729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:53.480558 32732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:53.480577 32730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:53.480861 32724 server_base.cc:1034] running on GCE node
I20260929 10:58:53.481089 32724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:58:53.481434 32724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:58:53.482604 32724 hybrid_clock.cc:648] HybridClock initialized: now 1790679533482597 us; error 34 us; skew 500 ppm
Sep 29 10:58:53 dist-test-slave-8nrt krb5kdc[32711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679533, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:58:53.485073 32724 init.cc:377] Logged in from keytab as kudu/127.10.238.254@KRBTEST.COM (short username kudu)
I20260929 10:58:53.485510 32724 webserver.cc:460] Webserver started at http://127.10.238.254:38287/ using document root <none> and password file <none>
I20260929 10:58:53.485702 32724 fs_manager.cc:359] Metadata directory not provided
I20260929 10:58:53.485744 32724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:58:53.485829 32724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:58:53.486789 32724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790679274510869-11195-0/minicluster-data/master-0/data/instance:
uuid: "35924787c7ac4f248950120dbb3f229d"
format_stamp: "Formatted at 2026-09-29 10:58:53 on dist-test-slave-8nrt"
server_key: "91205bde6694bd98e97d5ecaa8090126"
server_key_iv: "fbde91500d8bf86bbb8e06561cbac0b3"
server_key_version: "encryptionkey@0"
I20260929 10:58:53.487087 32724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790679274510869-11195-0/minicluster-data/master-0/wal/instance:
uuid: "35924787c7ac4f248950120dbb3f229d"
format_stamp: "Formatted at 2026-09-29 10:58:53 on dist-test-slave-8nrt"
server_key: "91205bde6694bd98e97d5ecaa8090126"
server_key_iv: "fbde91500d8bf86bbb8e06561cbac0b3"
server_key_version: "encryptionkey@0"
I20260929 10:58:53.488396 32724 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 10:58:53.489099 32739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:53.489300 32724 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:58:53.489379 32724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "35924787c7ac4f248950120dbb3f229d"
format_stamp: "Formatted at 2026-09-29 10:58:53 on dist-test-slave-8nrt"
server_key: "91205bde6694bd98e97d5ecaa8090126"
server_key_iv: "fbde91500d8bf86bbb8e06561cbac0b3"
server_key_version: "encryptionkey@0"
I20260929 10:58:53.489430 32724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:58:53.514647 32724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:58:53.516916 32724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:58:53.517025 32724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:58:53.520813 32724 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.254:36957
I20260929 10:58:53.520938 323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.254:36957 every 8 connection(s)
I20260929 10:58:53.521214 32724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:58:53.522111 324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:58:53.524305 324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35924787c7ac4f248950120dbb3f229d: Bootstrap starting.
I20260929 10:58:53.524871 324 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35924787c7ac4f248950120dbb3f229d: Neither blocks nor log segments found. Creating new log.
I20260929 10:58:53.525157 324 log.cc:824] T 00000000000000000000000000000000 P 35924787c7ac4f248950120dbb3f229d: Log is configured to *not* fsync() on all Append() calls
I20260929 10:58:53.525924 324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35924787c7ac4f248950120dbb3f229d: No bootstrap required, opened a new log
I20260929 10:58:53.527469 324 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35924787c7ac4f248950120dbb3f229d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35924787c7ac4f248950120dbb3f229d" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 36957 } }
I20260929 10:58:53.527626 324 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35924787c7ac4f248950120dbb3f229d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:58:53.527669 324 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35924787c7ac4f248950120dbb3f229d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35924787c7ac4f248950120dbb3f229d, State: Initialized, Role: FOLLOWER
I20260929 10:58:53.527787 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 32724
I20260929 10:58:53.527845 324 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35924787c7ac4f248950120dbb3f229d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35924787c7ac4f248950120dbb3f229d" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 36957 } }
I20260929 10:58:53.527909 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790679274510869-11195-0/minicluster-data/master-0/wal/instance
I20260929 10:58:53.527940 324 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35924787c7ac4f248950120dbb3f229d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:58:53.527976 324 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35924787c7ac4f248950120dbb3f229d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:58:53.528029 324 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35924787c7ac4f248950120dbb3f229d [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:58:53.528123 11195 external_mini_cluster.cc:1428] Setting key bb0a71f44cbe97b2c35774e082232b0c
I20260929 10:58:53.529083 324 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35924787c7ac4f248950120dbb3f229d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35924787c7ac4f248950120dbb3f229d" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 36957 } }
I20260929 10:58:53.529196 324 leader_election.cc:302] T 00000000000000000000000000000000 P 35924787c7ac4f248950120dbb3f229d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35924787c7ac4f248950120dbb3f229d; no voters:
I20260929 10:58:53.529376 324 leader_election.cc:288] T 00000000000000000000000000000000 P 35924787c7ac4f248950120dbb3f229d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:58:53.529436 329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35924787c7ac4f248950120dbb3f229d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:58:53.529578 329 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35924787c7ac4f248950120dbb3f229d [term 1 LEADER]: Becoming Leader. State: Replica: 35924787c7ac4f248950120dbb3f229d, State: Running, Role: LEADER
I20260929 10:58:53.529636 324 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35924787c7ac4f248950120dbb3f229d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:58:53.529660 329 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35924787c7ac4f248950120dbb3f229d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35924787c7ac4f248950120dbb3f229d" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 36957 } }
I20260929 10:58:53.531231 330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35924787c7ac4f248950120dbb3f229d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35924787c7ac4f248950120dbb3f229d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35924787c7ac4f248950120dbb3f229d" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 36957 } } }
I20260929 10:58:53.531333 330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35924787c7ac4f248950120dbb3f229d [sys.catalog]: This master's current role is: LEADER
I20260929 10:58:53.531675 331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35924787c7ac4f248950120dbb3f229d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35924787c7ac4f248950120dbb3f229d. Latest consensus state: current_term: 1 leader_uuid: "35924787c7ac4f248950120dbb3f229d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35924787c7ac4f248950120dbb3f229d" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 36957 } } }
I20260929 10:58:53.531698 343 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:58:53.531982 331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35924787c7ac4f248950120dbb3f229d [sys.catalog]: This master's current role is: LEADER
I20260929 10:58:53.532341 343 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:58:53.533804 343 catalog_manager.cc:1347] Generated new cluster ID: 0fad3315aac0446e9a3e2dd516571cd0
I20260929 10:58:53.533859 343 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 10:58:53 dist-test-slave-8nrt krb5kdc[32711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:58:53.538445 328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:53.528597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38445 (local address 127.10.238.254:36957)
0929 10:58:53.528796 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:53.528805 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:53.528842 (+ 37us) server_negotiation.cc:407] Connection header received
0929 10:58:53.528930 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:53.528935 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:53.528974 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:53.529079 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:53.529936 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:53.530984 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:53.531993 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:53.532550 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:53.532668 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:53.535197 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:53.535201 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:53.535202 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:58:53.535213 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:53.536849 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:53.537295 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:53.537306 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:53.537306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:53.537403 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:53.537643 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:53.537644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:53.537644 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:53.537873 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:53.538069 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:53.538151 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 10:58:53.538269 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":120,"threads_started":1}
I20260929 10:58:53.539376 11195 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:58:53.543948 343 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:58:53.544525 343 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:58:53.551092 328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:53.543758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47393 (local address 127.10.238.254:36957)
0929 10:58:53.543806 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:53.543808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:53.544045 (+ 237us) server_negotiation.cc:407] Connection header received
0929 10:58:53.544125 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:53.544126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:53.544136 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:53.544214 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:53.544861 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:53.545602 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:53.546606 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:53.546911 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:53.546973 (+ 62us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:53.548359 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:53.548360 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:53.548361 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:58:53.548367 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:53.549896 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:53.550307 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:53.550307 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:53.550308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:53.550361 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:53.550586 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:53.550586 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:53.550587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:53.550694 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:53.550799 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:53.550950 (+ 151us) server_negotiation.cc:299] Negotiation successful
0929 10:58:53.551008 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 10:58:53.560356 343 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35924787c7ac4f248950120dbb3f229d: Generated new TSK 0
I20260929 10:58:53.560557 343 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:58:53.563185 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 32724
W20260929 10:58:53.567468 32709 connection.cc:535] client connection to 127.10.238.254:36957 recv error: Network error: failed to read from TLS socket (remote: 127.10.238.254:36957): Connection reset by peer (error 104)
2026-09-29T10:58:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1333 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790679274510869-11195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:58:53 dist-test-slave-8nrt krb5kdc[356](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:58:53 dist-test-slave-8nrt krb5kdc[356](info): set up 2 sockets
Sep 29 10:58:53 dist-test-slave-8nrt krb5kdc[356](info): commencing operation
krb5kdc: starting...
W20260929 10:58:54.593385 11195 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Sep 29 10:58:54 dist-test-slave-8nrt krb5kdc[356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:58:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:58:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
I20260929 10:58:54.711531 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:34893
--webserver_interface=127.10.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:34449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:34893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:58:54.794185 369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:58:54.794502 369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:58:54.794543 369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:58:54.796168 369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:58:54.796227 369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:58:54.796259 369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:58:54.796278 369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:58:54.796303 369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:58:54.798151 369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:34449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790679274510869-11195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:34893
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:34893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:58:54.798349 369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:58:54.798511 369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:58:54.803248 375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:54.803273 377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:54.803313 374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:54.803493 369 server_base.cc:1034] running on GCE node
I20260929 10:58:54.803746 369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:58:54.804030 369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:58:54.805208 369 hybrid_clock.cc:648] HybridClock initialized: now 1790679534805201 us; error 41 us; skew 500 ppm
Sep 29 10:58:54 dist-test-slave-8nrt krb5kdc[356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679534, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:58:54.807641 369 init.cc:377] Logged in from keytab as kudu/127.10.238.254@KRBTEST.COM (short username kudu)
I20260929 10:58:54.808035 369 webserver.cc:460] Webserver started at http://127.10.238.254:41799/ using document root <none> and password file <none>
I20260929 10:58:54.808223 369 fs_manager.cc:359] Metadata directory not provided
I20260929 10:58:54.808264 369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:58:54.808349 369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:58:54.809441 369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790679274510869-11195-0/minicluster-data/master-0/data/instance:
uuid: "3d0858f9f08a4a5684a7ccb78eb6ca6f"
format_stamp: "Formatted at 2026-09-29 10:58:54 on dist-test-slave-8nrt"
server_key: "0612ed637f909774b4d1b56fe8edf622"
server_key_iv: "f6a9e6b8cd91b4dcb69b13e5a927d529"
server_key_version: "encryptionkey@0"
I20260929 10:58:54.809751 369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790679274510869-11195-0/minicluster-data/master-0/wal/instance:
uuid: "3d0858f9f08a4a5684a7ccb78eb6ca6f"
format_stamp: "Formatted at 2026-09-29 10:58:54 on dist-test-slave-8nrt"
server_key: "0612ed637f909774b4d1b56fe8edf622"
server_key_iv: "f6a9e6b8cd91b4dcb69b13e5a927d529"
server_key_version: "encryptionkey@0"
I20260929 10:58:54.811004 369 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 10:58:54.811683 384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:54.811899 369 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 10:58:54.812001 369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "3d0858f9f08a4a5684a7ccb78eb6ca6f"
format_stamp: "Formatted at 2026-09-29 10:58:54 on dist-test-slave-8nrt"
server_key: "0612ed637f909774b4d1b56fe8edf622"
server_key_iv: "f6a9e6b8cd91b4dcb69b13e5a927d529"
server_key_version: "encryptionkey@0"
I20260929 10:58:54.812091 369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:58:54.840906 369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:58:54.843308 369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:58:54.843422 369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:58:54.847425 369 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.254:34893
I20260929 10:58:54.847476 436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.254:34893 every 8 connection(s)
I20260929 10:58:54.847862 369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:58:54.848789 437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:58:54.851325 437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d0858f9f08a4a5684a7ccb78eb6ca6f: Bootstrap starting.
I20260929 10:58:54.851943 437 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d0858f9f08a4a5684a7ccb78eb6ca6f: Neither blocks nor log segments found. Creating new log.
I20260929 10:58:54.852238 437 log.cc:824] T 00000000000000000000000000000000 P 3d0858f9f08a4a5684a7ccb78eb6ca6f: Log is configured to *not* fsync() on all Append() calls
I20260929 10:58:54.852965 437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d0858f9f08a4a5684a7ccb78eb6ca6f: No bootstrap required, opened a new log
I20260929 10:58:54.854665 437 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d0858f9f08a4a5684a7ccb78eb6ca6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0858f9f08a4a5684a7ccb78eb6ca6f" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 34893 } }
I20260929 10:58:54.854809 437 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d0858f9f08a4a5684a7ccb78eb6ca6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:58:54.854864 437 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d0858f9f08a4a5684a7ccb78eb6ca6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d0858f9f08a4a5684a7ccb78eb6ca6f, State: Initialized, Role: FOLLOWER
I20260929 10:58:54.855036 437 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d0858f9f08a4a5684a7ccb78eb6ca6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0858f9f08a4a5684a7ccb78eb6ca6f" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 34893 } }
I20260929 10:58:54.855101 437 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d0858f9f08a4a5684a7ccb78eb6ca6f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:58:54.855142 437 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d0858f9f08a4a5684a7ccb78eb6ca6f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:58:54.855199 437 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d0858f9f08a4a5684a7ccb78eb6ca6f [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:58:54.856078 437 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d0858f9f08a4a5684a7ccb78eb6ca6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0858f9f08a4a5684a7ccb78eb6ca6f" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 34893 } }
I20260929 10:58:54.856200 437 leader_election.cc:302] T 00000000000000000000000000000000 P 3d0858f9f08a4a5684a7ccb78eb6ca6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d0858f9f08a4a5684a7ccb78eb6ca6f; no voters:
I20260929 10:58:54.856406 437 leader_election.cc:288] T 00000000000000000000000000000000 P 3d0858f9f08a4a5684a7ccb78eb6ca6f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:58:54.856437 440 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d0858f9f08a4a5684a7ccb78eb6ca6f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:58:54.856621 440 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d0858f9f08a4a5684a7ccb78eb6ca6f [term 1 LEADER]: Becoming Leader. State: Replica: 3d0858f9f08a4a5684a7ccb78eb6ca6f, State: Running, Role: LEADER
I20260929 10:58:54.856680 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 369
I20260929 10:58:54.856796 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790679274510869-11195-0/minicluster-data/master-0/wal/instance
I20260929 10:58:54.856772 437 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d0858f9f08a4a5684a7ccb78eb6ca6f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:58:54.856765 440 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d0858f9f08a4a5684a7ccb78eb6ca6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0858f9f08a4a5684a7ccb78eb6ca6f" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 34893 } }
I20260929 10:58:54.857008 11195 external_mini_cluster.cc:1428] Setting key 2c38c74955babd5e9efb9f45c2c7dc08
I20260929 10:58:54.857142 441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d0858f9f08a4a5684a7ccb78eb6ca6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d0858f9f08a4a5684a7ccb78eb6ca6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0858f9f08a4a5684a7ccb78eb6ca6f" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 34893 } } }
I20260929 10:58:54.857280 441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d0858f9f08a4a5684a7ccb78eb6ca6f [sys.catalog]: This master's current role is: LEADER
I20260929 10:58:54.857142 442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d0858f9f08a4a5684a7ccb78eb6ca6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d0858f9f08a4a5684a7ccb78eb6ca6f. Latest consensus state: current_term: 1 leader_uuid: "3d0858f9f08a4a5684a7ccb78eb6ca6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0858f9f08a4a5684a7ccb78eb6ca6f" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 34893 } } }
I20260929 10:58:54.857543 442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d0858f9f08a4a5684a7ccb78eb6ca6f [sys.catalog]: This master's current role is: LEADER
I20260929 10:58:54.858054 447 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:58:54.858589 447 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:58:54.860616 447 catalog_manager.cc:1347] Generated new cluster ID: c9726935a778450088d3319d18c9bd7a
I20260929 10:58:54.860671 447 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:58:54.863437 451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:54.857702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37029 (local address 127.10.238.254:34893)
0929 10:58:54.857930 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:54.857942 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:54.857981 (+ 39us) server_negotiation.cc:407] Connection header received
0929 10:58:54.858118 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:54.858125 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:54.858217 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:54.858496 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:54.858750 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:54.860100 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:54.861176 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:54.861863 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:54.861980 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:54.862720 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:54.862724 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:54.862724 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:58:54.862944 (+ 220us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:54.862985 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:58:54.863000 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:54.863086 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:54.863191 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 10:58:54.863225 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":83,"threads_started":1}
I20260929 10:58:54.864627 11195 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:58:54.865896 447 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:58:54.866750 447 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:58:54.869926 451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:54.866321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34517 (local address 127.10.238.254:34893)
0929 10:58:54.866358 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:54.866360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:54.866442 (+ 82us) server_negotiation.cc:407] Connection header received
0929 10:58:54.866504 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:54.866505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:54.866515 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:54.866572 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:54.866780 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:54.867520 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:54.868578 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:54.868864 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:54.868919 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:54.869669 (+ 750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:54.869671 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:54.869671 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:58:54.869675 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:54.869690 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:58:54.869694 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:54.869753 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:54.869846 (+ 93us) server_negotiation.cc:299] Negotiation successful
0929 10:58:54.869858 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 10:58:54.873333 447 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d0858f9f08a4a5684a7ccb78eb6ca6f: Generated new TSK 0
I20260929 10:58:54.873493 447 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:58:54.875524 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 369
W20260929 10:58:54.879551 354 connection.cc:535] client connection to 127.10.238.254:34893 recv error: Network error: failed to read from TLS socket (remote: 127.10.238.254:34893): Connection reset by peer (error 104)
2026-09-29T10:58:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1312 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790679274510869-11195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:58:54 dist-test-slave-8nrt krb5kdc[468](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:58:54 dist-test-slave-8nrt krb5kdc[468](info): set up 2 sockets
Sep 29 10:58:54 dist-test-slave-8nrt krb5kdc[468](info): commencing operation
krb5kdc: starting...
W20260929 10:58:55.929564 11195 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Sep 29 10:58:56 dist-test-slave-8nrt krb5kdc[468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:58:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:58:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
I20260929 10:58:56.045984 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:43441
--webserver_interface=127.10.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:40829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:43441
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:58:56.126544 481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:58:56.126762 481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:58:56.126787 481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:58:56.127948 481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:58:56.127990 481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:58:56.128005 481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:58:56.128017 481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:58:56.128077 481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:58:56.129594 481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:40829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790679274510869-11195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:43441
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:43441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:58:56.129859 481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:58:56.130095 481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:58:56.134961 489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:56.134948 486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:56.134971 487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:56.135192 481 server_base.cc:1034] running on GCE node
I20260929 10:58:56.135571 481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:58:56.135871 481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:58:56.137034 481 hybrid_clock.cc:648] HybridClock initialized: now 1790679536137022 us; error 31 us; skew 500 ppm
Sep 29 10:58:56 dist-test-slave-8nrt krb5kdc[468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679536, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:58:56.143296 481 init.cc:377] Logged in from keytab as kudu/127.10.238.254@KRBTEST.COM (short username kudu)
I20260929 10:58:56.143846 481 webserver.cc:460] Webserver started at http://127.10.238.254:40079/ using document root <none> and password file <none>
I20260929 10:58:56.144086 481 fs_manager.cc:359] Metadata directory not provided
I20260929 10:58:56.144166 481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:58:56.144327 481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:58:56.145526 481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790679274510869-11195-0/minicluster-data/master-0/data/instance:
uuid: "6bc314d0e072441d81b785b98a463915"
format_stamp: "Formatted at 2026-09-29 10:58:56 on dist-test-slave-8nrt"
server_key: "d73341e27a42e226ad28f7ab6e978ff2"
server_key_iv: "9b0c940da9aa6d4152d81119c64c87f5"
server_key_version: "encryptionkey@0"
I20260929 10:58:56.145897 481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790679274510869-11195-0/minicluster-data/master-0/wal/instance:
uuid: "6bc314d0e072441d81b785b98a463915"
format_stamp: "Formatted at 2026-09-29 10:58:56 on dist-test-slave-8nrt"
server_key: "d73341e27a42e226ad28f7ab6e978ff2"
server_key_iv: "9b0c940da9aa6d4152d81119c64c87f5"
server_key_version: "encryptionkey@0"
I20260929 10:58:56.147328 481 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 10:58:56.148217 496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:56.148430 481 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:58:56.148542 481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "6bc314d0e072441d81b785b98a463915"
format_stamp: "Formatted at 2026-09-29 10:58:56 on dist-test-slave-8nrt"
server_key: "d73341e27a42e226ad28f7ab6e978ff2"
server_key_iv: "9b0c940da9aa6d4152d81119c64c87f5"
server_key_version: "encryptionkey@0"
I20260929 10:58:56.148663 481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:58:56.152055 481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:58:56.154407 481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:58:56.154562 481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:58:56.158634 481 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.254:43441
I20260929 10:58:56.158672 548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.254:43441 every 8 connection(s)
I20260929 10:58:56.158982 481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:58:56.159940 549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:58:56.160588 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 481
I20260929 10:58:56.160755 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790679274510869-11195-0/minicluster-data/master-0/wal/instance
I20260929 10:58:56.160977 11195 external_mini_cluster.cc:1428] Setting key fd196bc85068c80c8702dd8144bda5d8
I20260929 10:58:56.163141 549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bc314d0e072441d81b785b98a463915: Bootstrap starting.
I20260929 10:58:56.163362 552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:56.161510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52643 (local address 127.10.238.254:43441)
0929 10:58:56.161680 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:56.161695 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:56.161720 (+ 25us) server_negotiation.cc:407] Connection header received
0929 10:58:56.162106 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:56.162125 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:56.162149 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:56.162260 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:56.162906 (+ 646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:56.162909 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:56.162909 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:58:56.162913 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:56.162949 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:58:56.162960 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:56.163014 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:56.163106 (+ 92us) server_negotiation.cc:299] Negotiation successful
0929 10:58:56.163169 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":96,"threads_started":1}
I20260929 10:58:56.163882 549 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bc314d0e072441d81b785b98a463915: Neither blocks nor log segments found. Creating new log.
I20260929 10:58:56.164215 549 log.cc:824] T 00000000000000000000000000000000 P 6bc314d0e072441d81b785b98a463915: Log is configured to *not* fsync() on all Append() calls
I20260929 10:58:56.164402 11195 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:58:56.164903 549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bc314d0e072441d81b785b98a463915: No bootstrap required, opened a new log
I20260929 10:58:56.167066 549 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bc314d0e072441d81b785b98a463915 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bc314d0e072441d81b785b98a463915" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 43441 } }
I20260929 10:58:56.167204 549 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6bc314d0e072441d81b785b98a463915 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:58:56.167227 549 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bc314d0e072441d81b785b98a463915 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bc314d0e072441d81b785b98a463915, State: Initialized, Role: FOLLOWER
I20260929 10:58:56.167371 549 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bc314d0e072441d81b785b98a463915 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bc314d0e072441d81b785b98a463915" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 43441 } }
I20260929 10:58:56.167457 549 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bc314d0e072441d81b785b98a463915 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:58:56.167501 549 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bc314d0e072441d81b785b98a463915 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:58:56.167550 549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bc314d0e072441d81b785b98a463915 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:58:56.168080 549 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bc314d0e072441d81b785b98a463915 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bc314d0e072441d81b785b98a463915" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 43441 } }
I20260929 10:58:56.168169 549 leader_election.cc:302] T 00000000000000000000000000000000 P 6bc314d0e072441d81b785b98a463915 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bc314d0e072441d81b785b98a463915; no voters:
I20260929 10:58:56.168359 549 leader_election.cc:288] T 00000000000000000000000000000000 P 6bc314d0e072441d81b785b98a463915 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:58:56.168404 559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bc314d0e072441d81b785b98a463915 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:58:56.168622 549 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bc314d0e072441d81b785b98a463915 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:58:56.168681 559 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bc314d0e072441d81b785b98a463915 [term 1 LEADER]: Becoming Leader. State: Replica: 6bc314d0e072441d81b785b98a463915, State: Running, Role: LEADER
I20260929 10:58:56.168780 559 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bc314d0e072441d81b785b98a463915 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bc314d0e072441d81b785b98a463915" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 43441 } }
I20260929 10:58:56.169130 560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bc314d0e072441d81b785b98a463915 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6bc314d0e072441d81b785b98a463915" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bc314d0e072441d81b785b98a463915" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 43441 } } }
I20260929 10:58:56.169134 561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bc314d0e072441d81b785b98a463915 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bc314d0e072441d81b785b98a463915. Latest consensus state: current_term: 1 leader_uuid: "6bc314d0e072441d81b785b98a463915" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bc314d0e072441d81b785b98a463915" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 43441 } } }
I20260929 10:58:56.169327 561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bc314d0e072441d81b785b98a463915 [sys.catalog]: This master's current role is: LEADER
I20260929 10:58:56.169314 560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bc314d0e072441d81b785b98a463915 [sys.catalog]: This master's current role is: LEADER
I20260929 10:58:56.169767 568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:58:56.170275 568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:58:56.171442 552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:56.165980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41131 (local address 127.10.238.254:43441)
0929 10:58:56.166014 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:56.166016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:56.170228 (+ 4212us) server_negotiation.cc:407] Connection header received
0929 10:58:56.170368 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:56.170369 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:56.170374 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:56.170407 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:56.171204 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:56.171205 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:56.171206 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:58:56.171210 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:56.171224 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:58:56.171229 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:56.171299 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:56.171359 (+ 60us) server_negotiation.cc:299] Negotiation successful
0929 10:58:56.171374 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 10:58:56.173668 568 catalog_manager.cc:1347] Generated new cluster ID: 48a9885f1c8f446b8eb1d9e069bbe52f
I20260929 10:58:56.173722 568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:58:56.195958 568 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:58:56.196151 568 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:58:56.211719 568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6bc314d0e072441d81b785b98a463915: Generated new TSK 0
I20260929 10:58:56.211941 568 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:58:56.216205 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 481
2026-09-29T10:58:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1341 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790679274510869-11195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:58:56 dist-test-slave-8nrt krb5kdc[580](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:58:56 dist-test-slave-8nrt krb5kdc[580](info): set up 2 sockets
Sep 29 10:58:56 dist-test-slave-8nrt krb5kdc[580](info): commencing operation
krb5kdc: starting...
W20260929 10:58:57.253456 11195 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Sep 29 10:58:57 dist-test-slave-8nrt krb5kdc[580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:58:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:58:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
I20260929 10:58:57.370393 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:39541
--webserver_interface=127.10.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:42981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:39541
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:58:57.451223 593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:58:57.451417 593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:58:57.451439 593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:58:57.452507 593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:58:57.452540 593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:58:57.452554 593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:58:57.452567 593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:58:57.452579 593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:58:57.454026 593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:42981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790679274510869-11195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:39541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:39541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:58:57.454216 593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:58:57.454376 593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:58:57.459256 599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:57.459302 598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:57.459262 601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:57.459304 593 server_base.cc:1034] running on GCE node
I20260929 10:58:57.459672 593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:58:57.459918 593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:58:57.461056 593 hybrid_clock.cc:648] HybridClock initialized: now 1790679537461045 us; error 27 us; skew 500 ppm
Sep 29 10:58:57 dist-test-slave-8nrt krb5kdc[580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679537, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:58:57.463685 593 init.cc:377] Logged in from keytab as kudu/127.10.238.254@KRBTEST.COM (short username kudu)
I20260929 10:58:57.464139 593 webserver.cc:460] Webserver started at http://127.10.238.254:34311/ using document root <none> and password file <none>
I20260929 10:58:57.464345 593 fs_manager.cc:359] Metadata directory not provided
I20260929 10:58:57.464385 593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:58:57.464471 593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:58:57.465616 593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790679274510869-11195-0/minicluster-data/master-0/data/instance:
uuid: "591774cda2a64f3cb121b3e24e36b762"
format_stamp: "Formatted at 2026-09-29 10:58:57 on dist-test-slave-8nrt"
server_key: "28686c5c8c593219d0b7248e50d734d5"
server_key_iv: "5a5a1290c9edc14ceffb63b4d810227c"
server_key_version: "encryptionkey@0"
I20260929 10:58:57.465924 593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790679274510869-11195-0/minicluster-data/master-0/wal/instance:
uuid: "591774cda2a64f3cb121b3e24e36b762"
format_stamp: "Formatted at 2026-09-29 10:58:57 on dist-test-slave-8nrt"
server_key: "28686c5c8c593219d0b7248e50d734d5"
server_key_iv: "5a5a1290c9edc14ceffb63b4d810227c"
server_key_version: "encryptionkey@0"
I20260929 10:58:57.467149 593 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:58:57.467833 608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:57.468004 593 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:58:57.468096 593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "591774cda2a64f3cb121b3e24e36b762"
format_stamp: "Formatted at 2026-09-29 10:58:57 on dist-test-slave-8nrt"
server_key: "28686c5c8c593219d0b7248e50d734d5"
server_key_iv: "5a5a1290c9edc14ceffb63b4d810227c"
server_key_version: "encryptionkey@0"
I20260929 10:58:57.468195 593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:58:57.476269 593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:58:57.478693 593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:58:57.478863 593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:58:57.483023 593 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.254:39541
I20260929 10:58:57.483057 660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.254:39541 every 8 connection(s)
I20260929 10:58:57.483464 593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:58:57.484542 661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:58:57.485230 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 593
I20260929 10:58:57.485595 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790679274510869-11195-0/minicluster-data/master-0/wal/instance
I20260929 10:58:57.485827 11195 external_mini_cluster.cc:1428] Setting key 02424676a6731833fa9d0ea47afd1eff
I20260929 10:58:57.487278 661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 591774cda2a64f3cb121b3e24e36b762: Bootstrap starting.
I20260929 10:58:57.488039 661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 591774cda2a64f3cb121b3e24e36b762: Neither blocks nor log segments found. Creating new log.
I20260929 10:58:57.488358 661 log.cc:824] T 00000000000000000000000000000000 P 591774cda2a64f3cb121b3e24e36b762: Log is configured to *not* fsync() on all Append() calls
I20260929 10:58:57.489121 661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 591774cda2a64f3cb121b3e24e36b762: No bootstrap required, opened a new log
I20260929 10:58:57.491704 661 raft_consensus.cc:348] T 00000000000000000000000000000000 P 591774cda2a64f3cb121b3e24e36b762 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "591774cda2a64f3cb121b3e24e36b762" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39541 } }
I20260929 10:58:57.491914 661 raft_consensus.cc:374] T 00000000000000000000000000000000 P 591774cda2a64f3cb121b3e24e36b762 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:58:57.491953 661 raft_consensus.cc:729] T 00000000000000000000000000000000 P 591774cda2a64f3cb121b3e24e36b762 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 591774cda2a64f3cb121b3e24e36b762, State: Initialized, Role: FOLLOWER
I20260929 10:58:57.492096 661 consensus_queue.cc:260] T 00000000000000000000000000000000 P 591774cda2a64f3cb121b3e24e36b762 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "591774cda2a64f3cb121b3e24e36b762" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39541 } }
I20260929 10:58:57.492194 661 raft_consensus.cc:388] T 00000000000000000000000000000000 P 591774cda2a64f3cb121b3e24e36b762 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:58:57.492231 661 raft_consensus.cc:482] T 00000000000000000000000000000000 P 591774cda2a64f3cb121b3e24e36b762 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:58:57.492285 661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 591774cda2a64f3cb121b3e24e36b762 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:58:57.493044 661 raft_consensus.cc:504] T 00000000000000000000000000000000 P 591774cda2a64f3cb121b3e24e36b762 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "591774cda2a64f3cb121b3e24e36b762" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39541 } }
I20260929 10:58:57.493132 661 leader_election.cc:302] T 00000000000000000000000000000000 P 591774cda2a64f3cb121b3e24e36b762 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 591774cda2a64f3cb121b3e24e36b762; no voters:
Sep 29 10:58:57 dist-test-slave-8nrt krb5kdc[580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:58:57.493362 661 leader_election.cc:288] T 00000000000000000000000000000000 P 591774cda2a64f3cb121b3e24e36b762 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:58:57.493384 666 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 591774cda2a64f3cb121b3e24e36b762 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:58:57.493552 666 raft_consensus.cc:686] T 00000000000000000000000000000000 P 591774cda2a64f3cb121b3e24e36b762 [term 1 LEADER]: Becoming Leader. State: Replica: 591774cda2a64f3cb121b3e24e36b762, State: Running, Role: LEADER
I20260929 10:58:57.493664 661 sys_catalog.cc:564] T 00000000000000000000000000000000 P 591774cda2a64f3cb121b3e24e36b762 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:58:57.493685 666 consensus_queue.cc:237] T 00000000000000000000000000000000 P 591774cda2a64f3cb121b3e24e36b762 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "591774cda2a64f3cb121b3e24e36b762" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39541 } }
I20260929 10:58:57.494134 667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 591774cda2a64f3cb121b3e24e36b762 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "591774cda2a64f3cb121b3e24e36b762" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "591774cda2a64f3cb121b3e24e36b762" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39541 } } }
I20260929 10:58:57.494227 667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 591774cda2a64f3cb121b3e24e36b762 [sys.catalog]: This master's current role is: LEADER
I20260929 10:58:57.494460 668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 591774cda2a64f3cb121b3e24e36b762 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 591774cda2a64f3cb121b3e24e36b762. Latest consensus state: current_term: 1 leader_uuid: "591774cda2a64f3cb121b3e24e36b762" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "591774cda2a64f3cb121b3e24e36b762" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39541 } } }
I20260929 10:58:57.494769 668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 591774cda2a64f3cb121b3e24e36b762 [sys.catalog]: This master's current role is: LEADER
I20260929 10:58:57.495565 673 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:58:57.496045 673 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:58:57.497609 673 catalog_manager.cc:1347] Generated new cluster ID: 7edffdf017d64c0383095a8872e7cd6e
I20260929 10:58:57.497660 673 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:58:57.498503 664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:57.486401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45711 (local address 127.10.238.254:39541)
0929 10:58:57.486571 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:57.486579 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:57.486605 (+ 26us) server_negotiation.cc:407] Connection header received
0929 10:58:57.487056 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:57.487063 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:57.487109 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:57.487231 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:57.488072 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:57.489353 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:57.490455 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:57.490894 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:57.490996 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:57.493935 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:57.493940 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:57.493940 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:58:57.493955 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:57.496429 (+ 2474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:57.496844 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:57.496847 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:57.496848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:57.496958 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:57.497175 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:57.497176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:57.497176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:57.497519 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:57.497671 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:57.498148 (+ 477us) server_negotiation.cc:299] Negotiation successful
0929 10:58:57.498301 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":82,"threads_started":1}
I20260929 10:58:57.499433 11195 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:58:57.508992 664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:57.500917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50539 (local address 127.10.238.254:39541)
0929 10:58:57.500953 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:57.500954 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:57.501533 (+ 579us) server_negotiation.cc:407] Connection header received
0929 10:58:57.501634 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:57.501635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:57.501645 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:57.501674 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:57.502411 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:57.503222 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:57.504283 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:57.504592 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:57.504635 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:57.506166 (+ 1531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:57.506167 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:57.506168 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:58:57.506175 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:57.507733 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:57.508132 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:57.508133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:57.508134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:57.508189 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:58:57.508401 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:58:57.508402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:58:57.508402 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:58:57.508535 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:57.508605 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:57.508815 (+ 210us) server_negotiation.cc:299] Negotiation successful
0929 10:58:57.508896 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 10:58:57.529793 673 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:58:57.530568 673 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:58:57.541082 673 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 591774cda2a64f3cb121b3e24e36b762: Generated new TSK 0
I20260929 10:58:57.541352 673 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:58:57.545701 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 593
2026-09-29T10:58:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1329 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790679274510869-11195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:58:57 dist-test-slave-8nrt krb5kdc[692](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:58:57 dist-test-slave-8nrt krb5kdc[692](info): set up 2 sockets
Sep 29 10:58:57 dist-test-slave-8nrt krb5kdc[692](info): commencing operation
krb5kdc: starting...
W20260929 10:58:58.569550 11195 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Sep 29 10:58:58 dist-test-slave-8nrt krb5kdc[692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:58:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:58:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
I20260929 10:58:58.686058 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:39927
--webserver_interface=127.10.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:43721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:39927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:58:58.765987 705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:58:58.766232 705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:58:58.766268 705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:58:58.767408 705 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:58:58.767475 705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:58:58.767500 705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:58:58.767521 705 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:58:58.767549 705 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:58:58.768993 705 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:43721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790679274510869-11195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:39927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:39927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:58:58.769284 705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:58:58.769507 705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:58:58.774334 710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:58.774353 705 server_base.cc:1034] running on GCE node
W20260929 10:58:58.774324 713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:58:58.774324 711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:58:58.774904 705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:58:58.775190 705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:58:58.776398 705 hybrid_clock.cc:648] HybridClock initialized: now 1790679538776383 us; error 41 us; skew 500 ppm
Sep 29 10:58:58 dist-test-slave-8nrt krb5kdc[692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679538, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:58:58.778861 705 init.cc:377] Logged in from keytab as kudu/127.10.238.254@KRBTEST.COM (short username kudu)
I20260929 10:58:58.779289 705 webserver.cc:460] Webserver started at http://127.10.238.254:37523/ using document root <none> and password file <none>
I20260929 10:58:58.779502 705 fs_manager.cc:359] Metadata directory not provided
I20260929 10:58:58.779554 705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:58:58.779693 705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:58:58.780761 705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790679274510869-11195-0/minicluster-data/master-0/data/instance:
uuid: "224721668c55470e8d1eb034a3cea48e"
format_stamp: "Formatted at 2026-09-29 10:58:58 on dist-test-slave-8nrt"
server_key: "0f442ccbc10230622cbeeb82db6da7b4"
server_key_iv: "63a95f49cb572f9a2717c62c5ddc6d69"
server_key_version: "encryptionkey@0"
I20260929 10:58:58.781117 705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790679274510869-11195-0/minicluster-data/master-0/wal/instance:
uuid: "224721668c55470e8d1eb034a3cea48e"
format_stamp: "Formatted at 2026-09-29 10:58:58 on dist-test-slave-8nrt"
server_key: "0f442ccbc10230622cbeeb82db6da7b4"
server_key_iv: "63a95f49cb572f9a2717c62c5ddc6d69"
server_key_version: "encryptionkey@0"
I20260929 10:58:58.782456 705 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 10:58:58.783231 720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:58:58.783398 705 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 10:58:58.783499 705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "224721668c55470e8d1eb034a3cea48e"
format_stamp: "Formatted at 2026-09-29 10:58:58 on dist-test-slave-8nrt"
server_key: "0f442ccbc10230622cbeeb82db6da7b4"
server_key_iv: "63a95f49cb572f9a2717c62c5ddc6d69"
server_key_version: "encryptionkey@0"
I20260929 10:58:58.783571 705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:58:58.796809 705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:58:58.799091 705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:58:58.799255 705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:58:58.803285 705 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.254:39927
I20260929 10:58:58.803330 772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.254:39927 every 8 connection(s)
I20260929 10:58:58.803691 705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:58:58.804559 773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:58:58.806869 773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 224721668c55470e8d1eb034a3cea48e: Bootstrap starting.
I20260929 10:58:58.807453 773 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 224721668c55470e8d1eb034a3cea48e: Neither blocks nor log segments found. Creating new log.
I20260929 10:58:58.807726 773 log.cc:824] T 00000000000000000000000000000000 P 224721668c55470e8d1eb034a3cea48e: Log is configured to *not* fsync() on all Append() calls
I20260929 10:58:58.808411 773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 224721668c55470e8d1eb034a3cea48e: No bootstrap required, opened a new log
I20260929 10:58:58.810058 773 raft_consensus.cc:348] T 00000000000000000000000000000000 P 224721668c55470e8d1eb034a3cea48e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "224721668c55470e8d1eb034a3cea48e" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39927 } }
I20260929 10:58:58.810194 773 raft_consensus.cc:374] T 00000000000000000000000000000000 P 224721668c55470e8d1eb034a3cea48e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:58:58.810216 773 raft_consensus.cc:729] T 00000000000000000000000000000000 P 224721668c55470e8d1eb034a3cea48e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 224721668c55470e8d1eb034a3cea48e, State: Initialized, Role: FOLLOWER
I20260929 10:58:58.810389 773 consensus_queue.cc:260] T 00000000000000000000000000000000 P 224721668c55470e8d1eb034a3cea48e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "224721668c55470e8d1eb034a3cea48e" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39927 } }
I20260929 10:58:58.810487 773 raft_consensus.cc:388] T 00000000000000000000000000000000 P 224721668c55470e8d1eb034a3cea48e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:58:58.810524 773 raft_consensus.cc:482] T 00000000000000000000000000000000 P 224721668c55470e8d1eb034a3cea48e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:58:58.810580 773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 224721668c55470e8d1eb034a3cea48e [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:58:58.810655 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 705
I20260929 10:58:58.810770 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790679274510869-11195-0/minicluster-data/master-0/wal/instance
I20260929 10:58:58.810958 11195 external_mini_cluster.cc:1428] Setting key 256e06e1eb281a480694c1a8f1478d9e
I20260929 10:58:58.811764 773 raft_consensus.cc:504] T 00000000000000000000000000000000 P 224721668c55470e8d1eb034a3cea48e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "224721668c55470e8d1eb034a3cea48e" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39927 } }
I20260929 10:58:58.811851 773 leader_election.cc:302] T 00000000000000000000000000000000 P 224721668c55470e8d1eb034a3cea48e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 224721668c55470e8d1eb034a3cea48e; no voters:
I20260929 10:58:58.812034 773 leader_election.cc:288] T 00000000000000000000000000000000 P 224721668c55470e8d1eb034a3cea48e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:58:58.812054 778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 224721668c55470e8d1eb034a3cea48e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:58:58.812219 778 raft_consensus.cc:686] T 00000000000000000000000000000000 P 224721668c55470e8d1eb034a3cea48e [term 1 LEADER]: Becoming Leader. State: Replica: 224721668c55470e8d1eb034a3cea48e, State: Running, Role: LEADER
I20260929 10:58:58.812359 773 sys_catalog.cc:564] T 00000000000000000000000000000000 P 224721668c55470e8d1eb034a3cea48e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:58:58.812366 778 consensus_queue.cc:237] T 00000000000000000000000000000000 P 224721668c55470e8d1eb034a3cea48e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "224721668c55470e8d1eb034a3cea48e" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39927 } }
I20260929 10:58:58.812853 779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 224721668c55470e8d1eb034a3cea48e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 224721668c55470e8d1eb034a3cea48e. Latest consensus state: current_term: 1 leader_uuid: "224721668c55470e8d1eb034a3cea48e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "224721668c55470e8d1eb034a3cea48e" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39927 } } }
I20260929 10:58:58.812963 779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 224721668c55470e8d1eb034a3cea48e [sys.catalog]: This master's current role is: LEADER
I20260929 10:58:58.813563 780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 224721668c55470e8d1eb034a3cea48e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "224721668c55470e8d1eb034a3cea48e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "224721668c55470e8d1eb034a3cea48e" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 39927 } } }
I20260929 10:58:58.813704 780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 224721668c55470e8d1eb034a3cea48e [sys.catalog]: This master's current role is: LEADER
I20260929 10:58:58.813841 787 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:58:58.814406 787 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:58:58.816972 777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:58.811457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41703 (local address 127.10.238.254:39927)
0929 10:58:58.811637 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:58.811646 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:58.811704 (+ 58us) server_negotiation.cc:407] Connection header received
0929 10:58:58.811800 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:58.811806 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:58.811877 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:58.812025 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:58.812279 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:58.813715 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:58.814881 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:58.815503 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:58.815609 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:58.816278 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:58.816281 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:58.816281 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:58:58.816500 (+ 219us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:58.816539 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:58:58.816555 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:58.816627 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:58.816694 (+ 67us) server_negotiation.cc:299] Negotiation successful
0929 10:58:58.816761 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":98,"threads_started":1}
I20260929 10:58:58.817962 11195 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:58:58.821700 787 catalog_manager.cc:1347] Generated new cluster ID: d38098d251f4418e83b825e17ed71d41
I20260929 10:58:58.821766 787 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:58:58.822654 777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:58:58.819487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35471 (local address 127.10.238.254:39927)
0929 10:58:58.819526 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 10:58:58.819527 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 10:58:58.819621 (+ 94us) server_negotiation.cc:407] Connection header received
0929 10:58:58.819678 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:58:58.819679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:58:58.819691 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:58:58.819753 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:58:58.819842 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:58.820582 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:58.821249 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:58:58.821636 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:58:58.821709 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:58:58.822411 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:58:58.822412 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:58:58.822412 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:58:58.822417 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:58:58.822432 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:58:58.822437 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:58:58.822492 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 10:58:58.822552 (+ 60us) server_negotiation.cc:299] Negotiation successful
0929 10:58:58.822576 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 10:58:58.829538 787 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:58:58.830086 787 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:58:58.832831 787 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 224721668c55470e8d1eb034a3cea48e: Generated new TSK 0
I20260929 10:58:58.832996 787 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:58:58.837450 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 705
2026-09-29T10:58:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1291 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790679274510869-11195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:58:58 dist-test-slave-8nrt krb5kdc[804](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:58:58 dist-test-slave-8nrt krb5kdc[804](info): set up 2 sockets
Sep 29 10:58:58 dist-test-slave-8nrt krb5kdc[804](info): commencing operation
krb5kdc: starting...
W20260929 10:58:59.869807 11195 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Sep 29 10:58:59 dist-test-slave-8nrt krb5kdc[804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:58:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:58:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
I20260929 10:58:59.991938 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:34753
--webserver_interface=127.10.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:45443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:34753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:59:00.072459 817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:59:00.072674 817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:59:00.072697 817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:59:00.073865 817 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:59:00.073909 817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:59:00.073923 817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:59:00.073935 817 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:59:00.073947 817 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:59:00.075428 817 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:45443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790679274510869-11195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:34753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:34753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:59:00.075687 817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:59:00.075923 817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:59:00.080830 825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:59:00.080838 823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:59:00.080838 822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:59:00.080919 817 server_base.cc:1034] running on GCE node
I20260929 10:59:00.081353 817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:59:00.081641 817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:59:00.082839 817 hybrid_clock.cc:648] HybridClock initialized: now 1790679540082833 us; error 41 us; skew 500 ppm
Sep 29 10:59:00 dist-test-slave-8nrt krb5kdc[804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679540, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:59:00.085498 817 init.cc:377] Logged in from keytab as kudu/127.10.238.254@KRBTEST.COM (short username kudu)
I20260929 10:59:00.085948 817 webserver.cc:460] Webserver started at http://127.10.238.254:46465/ using document root <none> and password file <none>
I20260929 10:59:00.086174 817 fs_manager.cc:359] Metadata directory not provided
I20260929 10:59:00.086222 817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:59:00.086377 817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:59:00.087407 817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790679274510869-11195-0/minicluster-data/master-0/data/instance:
uuid: "778d6a6a7bf24a569c3c301724cdc17e"
format_stamp: "Formatted at 2026-09-29 10:59:00 on dist-test-slave-8nrt"
server_key: "546f3019ae0eb8b9703182ca26cec1b4"
server_key_iv: "d2baf31f22fe5496a5dc21c1764fc6ce"
server_key_version: "encryptionkey@0"
I20260929 10:59:00.087843 817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790679274510869-11195-0/minicluster-data/master-0/wal/instance:
uuid: "778d6a6a7bf24a569c3c301724cdc17e"
format_stamp: "Formatted at 2026-09-29 10:59:00 on dist-test-slave-8nrt"
server_key: "546f3019ae0eb8b9703182ca26cec1b4"
server_key_iv: "d2baf31f22fe5496a5dc21c1764fc6ce"
server_key_version: "encryptionkey@0"
I20260929 10:59:00.089184 817 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 10:59:00.089970 832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:59:00.090148 817 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:59:00.090247 817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "778d6a6a7bf24a569c3c301724cdc17e"
format_stamp: "Formatted at 2026-09-29 10:59:00 on dist-test-slave-8nrt"
server_key: "546f3019ae0eb8b9703182ca26cec1b4"
server_key_iv: "d2baf31f22fe5496a5dc21c1764fc6ce"
server_key_version: "encryptionkey@0"
I20260929 10:59:00.090327 817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:59:00.093637 817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:59:00.095504 817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:59:00.095657 817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:59:00.099539 817 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.254:34753
I20260929 10:59:00.099570 884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.254:34753 every 8 connection(s)
I20260929 10:59:00.099897 817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:59:00.100899 885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:59:00.103245 885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 778d6a6a7bf24a569c3c301724cdc17e: Bootstrap starting.
I20260929 10:59:00.103785 885 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 778d6a6a7bf24a569c3c301724cdc17e: Neither blocks nor log segments found. Creating new log.
I20260929 10:59:00.104048 885 log.cc:824] T 00000000000000000000000000000000 P 778d6a6a7bf24a569c3c301724cdc17e: Log is configured to *not* fsync() on all Append() calls
I20260929 10:59:00.104736 885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 778d6a6a7bf24a569c3c301724cdc17e: No bootstrap required, opened a new log
I20260929 10:59:00.106479 885 raft_consensus.cc:348] T 00000000000000000000000000000000 P 778d6a6a7bf24a569c3c301724cdc17e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778d6a6a7bf24a569c3c301724cdc17e" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 34753 } }
I20260929 10:59:00.106667 885 raft_consensus.cc:374] T 00000000000000000000000000000000 P 778d6a6a7bf24a569c3c301724cdc17e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:59:00.106726 885 raft_consensus.cc:729] T 00000000000000000000000000000000 P 778d6a6a7bf24a569c3c301724cdc17e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 778d6a6a7bf24a569c3c301724cdc17e, State: Initialized, Role: FOLLOWER
I20260929 10:59:00.106851 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 817
I20260929 10:59:00.106885 885 consensus_queue.cc:260] T 00000000000000000000000000000000 P 778d6a6a7bf24a569c3c301724cdc17e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778d6a6a7bf24a569c3c301724cdc17e" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 34753 } }
I20260929 10:59:00.106968 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790679274510869-11195-0/minicluster-data/master-0/wal/instance
I20260929 10:59:00.106977 885 raft_consensus.cc:388] T 00000000000000000000000000000000 P 778d6a6a7bf24a569c3c301724cdc17e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:59:00.107021 885 raft_consensus.cc:482] T 00000000000000000000000000000000 P 778d6a6a7bf24a569c3c301724cdc17e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:59:00.107070 885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 778d6a6a7bf24a569c3c301724cdc17e [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:59:00.107163 11195 external_mini_cluster.cc:1428] Setting key 7e451a33842492935a1ba8e00ce4eb9e
I20260929 10:59:00.108141 885 raft_consensus.cc:504] T 00000000000000000000000000000000 P 778d6a6a7bf24a569c3c301724cdc17e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778d6a6a7bf24a569c3c301724cdc17e" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 34753 } }
I20260929 10:59:00.108259 885 leader_election.cc:302] T 00000000000000000000000000000000 P 778d6a6a7bf24a569c3c301724cdc17e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 778d6a6a7bf24a569c3c301724cdc17e; no voters:
I20260929 10:59:00.108775 885 leader_election.cc:288] T 00000000000000000000000000000000 P 778d6a6a7bf24a569c3c301724cdc17e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:59:00.108835 890 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 778d6a6a7bf24a569c3c301724cdc17e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:59:00.108975 890 raft_consensus.cc:686] T 00000000000000000000000000000000 P 778d6a6a7bf24a569c3c301724cdc17e [term 1 LEADER]: Becoming Leader. State: Replica: 778d6a6a7bf24a569c3c301724cdc17e, State: Running, Role: LEADER
I20260929 10:59:00.109074 885 sys_catalog.cc:564] T 00000000000000000000000000000000 P 778d6a6a7bf24a569c3c301724cdc17e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:59:00.109239 890 consensus_queue.cc:237] T 00000000000000000000000000000000 P 778d6a6a7bf24a569c3c301724cdc17e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778d6a6a7bf24a569c3c301724cdc17e" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 34753 } }
I20260929 10:59:00.109479 889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:00.107648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40819 (local address 127.10.238.254:34753)
0929 10:59:00.107852 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:00.107861 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:00.107893 (+ 32us) server_negotiation.cc:407] Connection header received
0929 10:59:00.107983 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:00.107988 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:00.108009 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:00.108115 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:00.108858 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:59:00.108863 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:59:00.108863 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:59:00.108867 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:59:00.108911 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:59:00.108928 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:59:00.108997 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 10:59:00.109175 (+ 178us) server_negotiation.cc:299] Negotiation successful
0929 10:59:00.109218 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":126,"threads_started":1}
I20260929 10:59:00.109661 891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 778d6a6a7bf24a569c3c301724cdc17e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "778d6a6a7bf24a569c3c301724cdc17e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778d6a6a7bf24a569c3c301724cdc17e" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 34753 } } }
I20260929 10:59:00.109733 891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 778d6a6a7bf24a569c3c301724cdc17e [sys.catalog]: This master's current role is: LEADER
I20260929 10:59:00.109941 899 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:59:00.109982 892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 778d6a6a7bf24a569c3c301724cdc17e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 778d6a6a7bf24a569c3c301724cdc17e. Latest consensus state: current_term: 1 leader_uuid: "778d6a6a7bf24a569c3c301724cdc17e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778d6a6a7bf24a569c3c301724cdc17e" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 34753 } } }
I20260929 10:59:00.110129 892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 778d6a6a7bf24a569c3c301724cdc17e [sys.catalog]: This master's current role is: LEADER
I20260929 10:59:00.110488 899 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:59:00.110512 11195 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:59:00.112094 899 catalog_manager.cc:1347] Generated new cluster ID: 60595287d0774e6dbbc8df19af49546b
I20260929 10:59:00.112149 899 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:59:00.114624 889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:00.112018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40053 (local address 127.10.238.254:34753)
0929 10:59:00.112054 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:00.112055 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:00.113572 (+ 1517us) server_negotiation.cc:407] Connection header received
0929 10:59:00.113656 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:00.113657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:00.113672 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:00.113732 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:00.114434 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:59:00.114435 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:59:00.114436 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:59:00.114438 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:59:00.114451 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:59:00.114454 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:59:00.114491 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0929 10:59:00.114554 (+ 63us) server_negotiation.cc:299] Negotiation successful
0929 10:59:00.114566 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 10:59:00.117348 899 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:59:00.117491 899 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:59:00.119479 899 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 778d6a6a7bf24a569c3c301724cdc17e: Generated new TSK 0
I20260929 10:59:00.119661 899 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:59:00.122618 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 817
2026-09-29T10:59:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1284 ms)
[----------] 18 tests from AuthTokenIssuingTest (23813 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:59:00 dist-test-slave-8nrt krb5kdc[916](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:59:00 dist-test-slave-8nrt krb5kdc[916](info): set up 2 sockets
Sep 29 10:59:00 dist-test-slave-8nrt krb5kdc[916](info): commencing operation
krb5kdc: starting...
W20260929 10:59:01.193581 11195 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Sep 29 10:59:01 dist-test-slave-8nrt krb5kdc[916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:59:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:59:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
I20260929 10:59:01.313450 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:33259
--webserver_interface=127.10.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:39561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:33259,127.10.238.253:33389,127.10.238.252:39755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:59:01.393697 929 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 10:59:01.393947 929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:59:01.393978 929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:59:01.394019 929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:59:01.395205 929 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:59:01.395268 929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:59:01.395292 929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:59:01.395320 929 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:59:01.395349 929 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:59:01.396966 929 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:39561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:33259,127.10.238.253:33389,127.10.238.252:39755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:33259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:59:01.397235 929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:59:01.397555 929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:59:01.402441 937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:59:01.402405 934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:59:01.402702 929 server_base.cc:1034] running on GCE node
W20260929 10:59:01.402444 935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:59:01.403017 929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:59:01.403257 929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:59:01.404425 929 hybrid_clock.cc:648] HybridClock initialized: now 1790679541404420 us; error 33 us; skew 500 ppm
Sep 29 10:59:01 dist-test-slave-8nrt krb5kdc[916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679541, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:59:01.416972 929 init.cc:377] Logged in from keytab as kudu/127.10.238.254@KRBTEST.COM (short username kudu)
I20260929 10:59:01.417481 929 webserver.cc:460] Webserver started at http://127.10.238.254:43437/ using document root <none> and password file <none>
I20260929 10:59:01.417704 929 fs_manager.cc:359] Metadata directory not provided
I20260929 10:59:01.417748 929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:59:01.417884 929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:59:01.418948 929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/minicluster-data/master-0/data/instance:
uuid: "4cec287e97e04da2abd963f533be8cdb"
format_stamp: "Formatted at 2026-09-29 10:59:01 on dist-test-slave-8nrt"
server_key: "7a60dc1b45b35ff7b063af1a2f2017c3"
server_key_iv: "6f285d0e068110acd0fd9340b6e53b2d"
server_key_version: "encryptionkey@0"
I20260929 10:59:01.419304 929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/minicluster-data/master-0/wal/instance:
uuid: "4cec287e97e04da2abd963f533be8cdb"
format_stamp: "Formatted at 2026-09-29 10:59:01 on dist-test-slave-8nrt"
server_key: "7a60dc1b45b35ff7b063af1a2f2017c3"
server_key_iv: "6f285d0e068110acd0fd9340b6e53b2d"
server_key_version: "encryptionkey@0"
I20260929 10:59:01.420642 929 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 10:59:01.421430 944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:59:01.421602 929 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:59:01.421729 929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "4cec287e97e04da2abd963f533be8cdb"
format_stamp: "Formatted at 2026-09-29 10:59:01 on dist-test-slave-8nrt"
server_key: "7a60dc1b45b35ff7b063af1a2f2017c3"
server_key_iv: "6f285d0e068110acd0fd9340b6e53b2d"
server_key_version: "encryptionkey@0"
I20260929 10:59:01.421808 929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:59:01.434190 929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:59:01.436368 929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:59:01.436522 929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:59:01.440632 929 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.254:33259
I20260929 10:59:01.440686 996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.254:33259 every 8 connection(s)
I20260929 10:59:01.441035 929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:59:01.441915 997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:59:01.443467 997 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.238.254" port: 33259 } has no permanent_uuid. Determining permanent_uuid...
I20260929 10:59:01.448136 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 929
I20260929 10:59:01.448266 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/minicluster-data/master-0/wal/instance
I20260929 10:59:01.448467 11195 external_mini_cluster.cc:1428] Setting key 504af6316f9975dd9a498530050a3de9
Sep 29 10:59:01 dist-test-slave-8nrt krb5kdc[916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679541, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:59:01.453349 1000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:01.444203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50530 (local address 127.10.238.254:33259)
0929 10:59:01.444348 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:01.444361 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:01.444450 (+ 89us) server_negotiation.cc:407] Connection header received
0929 10:59:01.444937 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:01.444939 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:01.444955 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:01.445039 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:01.445628 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:01.446431 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:01.447324 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:01.447694 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:01.447796 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:01.450068 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:59:01.450075 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:59:01.450076 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:59:01.450086 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:59:01.451599 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:01.452087 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:01.452091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:01.452091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:01.452196 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:01.452458 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:01.452459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:01.452459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:01.452697 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:59:01.452768 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 10:59:01.453028 (+ 260us) server_negotiation.cc:299] Negotiation successful
0929 10:59:01.453123 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":80,"threads_started":1}
I20260929 10:59:01.453450 999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:01.444136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:33259 (local address 127.0.0.1:50530)
0929 10:59:01.444314 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 10:59:01.444353 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 10:59:01.444816 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:59:01.445064 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:59:01.445066 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:59:01.445386 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:59:01.445542 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:59:01.445546 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:59:01.446535 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:59:01.446536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:59:01.447220 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:59:01.447222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:59:01.447803 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:59:01.447804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:59:01.447886 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:01.448466 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:59:01.448473 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:59:01.449966 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:59:01.451683 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:59:01.451691 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:59:01.451693 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:59:01.451974 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:59:01.452272 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:59:01.452273 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:59:01.452273 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:59:01.452386 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:59:01.452772 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:59:01.452779 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:59:01.452861 (+ 82us) client_negotiation.cc:769] Sending connection context
0929 10:59:01.452976 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 10:59:01.453238 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":79,"thread_start_us":101,"threads_started":1}
I20260929 10:59:01.454406 997 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.238.253" port: 33389 } has no permanent_uuid. Determining permanent_uuid...
I20260929 10:59:01.454752 999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:01.454620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.253:33389 (local address 127.0.0.1:53990)
0929 10:59:01.454671 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0929 10:59:01.454724 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.253:33389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260929 10:59:01.454911 947 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.238.253:33389: connect: Connection refused (error 111)
I20260929 10:59:01.455356 999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:01.455254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.253:33389 (local address 127.0.0.1:54000)
0929 10:59:01.455301 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0929 10:59:01.455329 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.253:33389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260929 10:59:01.455492 997 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.238.253:33389: Network error: Client connection negotiation failed: client connection to 127.10.238.253:33389: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.10.238.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.253.keytab.
Entry for principal kudu/127.10.238.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.253.keytab.
I20260929 10:59:01.472244 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.253:33389
--webserver_interface=127.10.238.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.253.keytab
--principal=kudu/127.10.238.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:39561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:33259,127.10.238.253:33389,127.10.238.252:39755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:59:01.504733 997 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.238.253" port: 33389 } attempt: 1
I20260929 10:59:01.505342 999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:01.505145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.253:33389 (local address 127.0.0.1:54008)
0929 10:59:01.505216 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0929 10:59:01.505308 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.253:33389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 10:59:01.505689 999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:01.505611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.253:33389 (local address 127.0.0.1:54016)
0929 10:59:01.505642 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0929 10:59:01.505670 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.253:33389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260929 10:59:01.505800 997 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.238.253:33389: Network error: Client connection negotiation failed: client connection to 127.10.238.253:33389: connect: Connection refused (error 111)
W20260929 10:59:01.552799 1004 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 10:59:01.553037 1004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:59:01.553067 1004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:59:01.553108 1004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:59:01.554260 1004 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:59:01.554322 1004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:59:01.554345 1004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:59:01.554374 1004 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:59:01.554397 1004 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:59:01.555835 1004 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:39561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:33259,127.10.238.253:33389,127.10.238.252:39755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.253.keytab
--principal=kudu/127.10.238.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.253:33389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.238.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:59:01.556102 1004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:59:01.556339 1004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:59:01.560951 1010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:59:01.560943 1009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:59:01.561069 1004 server_base.cc:1034] running on GCE node
W20260929 10:59:01.560979 1012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:59:01.561506 1004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:59:01.561825 1004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:59:01.563124 1004 hybrid_clock.cc:648] HybridClock initialized: now 1790679541563112 us; error 40 us; skew 500 ppm
Sep 29 10:59:01 dist-test-slave-8nrt krb5kdc[916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679541, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:59:01.565784 1004 init.cc:377] Logged in from keytab as kudu/127.10.238.253@KRBTEST.COM (short username kudu)
I20260929 10:59:01.566226 1004 webserver.cc:460] Webserver started at http://127.10.238.253:45299/ using document root <none> and password file <none>
I20260929 10:59:01.566401 1004 fs_manager.cc:359] Metadata directory not provided
I20260929 10:59:01.566440 1004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:59:01.566529 1004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:59:01.567526 1004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/minicluster-data/master-1/data/instance:
uuid: "f751508dffee4676861de0d1074550d7"
format_stamp: "Formatted at 2026-09-29 10:59:01 on dist-test-slave-8nrt"
server_key: "18bd40f6dfa4197b9d6786bfe9ebe778"
server_key_iv: "1e8e73babcbaf3327d4fe1b2fe736473"
server_key_version: "encryptionkey@0"
I20260929 10:59:01.567832 1004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/minicluster-data/master-1/wal/instance:
uuid: "f751508dffee4676861de0d1074550d7"
format_stamp: "Formatted at 2026-09-29 10:59:01 on dist-test-slave-8nrt"
server_key: "18bd40f6dfa4197b9d6786bfe9ebe778"
server_key_iv: "1e8e73babcbaf3327d4fe1b2fe736473"
server_key_version: "encryptionkey@0"
I20260929 10:59:01.569072 1004 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 10:59:01.569804 1019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:59:01.569979 1004 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 10:59:01.570067 1004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/minicluster-data/master-1/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/minicluster-data/master-1/wal
uuid: "f751508dffee4676861de0d1074550d7"
format_stamp: "Formatted at 2026-09-29 10:59:01 on dist-test-slave-8nrt"
server_key: "18bd40f6dfa4197b9d6786bfe9ebe778"
server_key_iv: "1e8e73babcbaf3327d4fe1b2fe736473"
server_key_version: "encryptionkey@0"
I20260929 10:59:01.570137 1004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:59:01.573942 997 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.238.253" port: 33389 } attempt: 2
I20260929 10:59:01.574323 999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:01.574188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.253:33389 (local address 127.0.0.1:54032)
0929 10:59:01.574246 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0929 10:59:01.574287 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.253:33389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260929 10:59:01.574672 999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:01.574582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.253:33389 (local address 127.0.0.1:54042)
0929 10:59:01.574623 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0929 10:59:01.574649 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.253:33389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260929 10:59:01.574783 997 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.238.253:33389: Network error: Client connection negotiation failed: client connection to 127.10.238.253:33389: connect: Connection refused (error 111)
I20260929 10:59:01.581090 1004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:59:01.582872 1004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:59:01.583019 1004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:59:01.586925 1004 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.253:33389
I20260929 10:59:01.586968 1071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.253:33389 every 8 connection(s)
I20260929 10:59:01.587296 1004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/minicluster-data/master-1/data/info.pb
I20260929 10:59:01.588176 1072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:59:01.589733 1072 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.238.254" port: 33259 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 10:59:01 dist-test-slave-8nrt krb5kdc[916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679541, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.253@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:59:01.596858 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 1004
I20260929 10:59:01.596968 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/minicluster-data/master-1/wal/instance
I20260929 10:59:01.597150 11195 external_mini_cluster.cc:1428] Setting key 32976adcf58e3351b74dac95c3c1cd52
I20260929 10:59:01.598574 1000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:01.590375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50536 (local address 127.10.238.254:33259)
0929 10:59:01.590436 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:01.590439 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:01.590650 (+ 211us) server_negotiation.cc:407] Connection header received
0929 10:59:01.591110 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:01.591111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:01.591124 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:01.591185 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:01.591744 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:01.592208 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:01.593141 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:01.593379 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:01.593433 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:01.595405 (+ 1972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:59:01.595407 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:59:01.595407 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:59:01.595413 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:59:01.596948 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:01.597493 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:01.597494 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:01.597494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:01.597554 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:01.597873 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:01.597874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:01.597875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:01.598023 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:59:01.598093 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 10:59:01.598342 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 10:59:01.598437 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 10:59:01.598666 1074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:01.590360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:33259 (local address 127.0.0.1:50536)
0929 10:59:01.590539 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 10:59:01.590575 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 10:59:01.591028 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:59:01.591249 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:59:01.591252 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:59:01.591491 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:59:01.591676 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:59:01.591680 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:59:01.592290 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:59:01.592291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:59:01.593069 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:59:01.593071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:59:01.593449 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:59:01.593450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:59:01.593550 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:01.593991 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:59:01.594006 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:59:01.595323 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:59:01.597067 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:59:01.597076 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:59:01.597078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:59:01.597415 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:59:01.597669 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:59:01.597670 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:59:01.597671 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:59:01.597795 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:59:01.598094 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:59:01.598101 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:59:01.598216 (+ 115us) client_negotiation.cc:769] Sending connection context
0929 10:59:01.598305 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 10:59:01.598429 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":104,"threads_started":1}
I20260929 10:59:01.599215 1072 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.238.253" port: 33389 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 10:59:01 dist-test-slave-8nrt krb5kdc[916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679541, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.253@KRBTEST.COM for kudu/127.10.238.253@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.238.252@KRBTEST.COM; defaulting to no policy
I20260929 10:59:01.606530 1074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:01.599459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.253:33389 (local address 127.0.0.1:54046)
0929 10:59:01.599504 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0929 10:59:01.599513 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 10:59:01.599547 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:59:01.599909 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:59:01.599910 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:59:01.600045 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:59:01.600105 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:59:01.600106 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:59:01.601063 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:59:01.601064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:59:01.601628 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:59:01.601630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:59:01.602102 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:59:01.602103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:59:01.602175 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:01.602572 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:59:01.602577 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:59:01.603690 (+ 1113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:59:01.605432 (+ 1742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:59:01.605433 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:59:01.605433 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:59:01.605590 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:59:01.605858 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:59:01.605859 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:59:01.605859 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:59:01.605919 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:59:01.606231 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:59:01.606232 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:59:01.606281 (+ 49us) client_negotiation.cc:769] Sending connection context
0929 10:59:01.606344 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 10:59:01.606425 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260929 10:59:01.606560 1076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:01.599542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54046 (local address 127.10.238.253:33389)
0929 10:59:01.599742 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:01.599746 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:01.599764 (+ 18us) server_negotiation.cc:407] Connection header received
0929 10:59:01.599783 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:01.599784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:01.599795 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:01.599905 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:01.600200 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:01.600970 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:01.601717 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:01.602017 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:01.602102 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:01.603769 (+ 1667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:59:01.603778 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:59:01.603778 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:59:01.603785 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:59:01.605348 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:01.605691 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:01.605695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:01.605695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:01.605770 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:01.605987 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:01.605988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:01.605989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:01.606144 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:59:01.606210 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 10:59:01.606365 (+ 155us) server_negotiation.cc:299] Negotiation successful
0929 10:59:01.606472 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":164,"thread_start_us":109,"threads_started":1}
I20260929 10:59:01.607250 1072 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.238.252" port: 39755 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.252@KRBTEST.COM" created.
I20260929 10:59:01.607602 1074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:01.607470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.252:39755 (local address 127.0.0.1:47666)
0929 10:59:01.607517 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0929 10:59:01.607570 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.252:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260929 10:59:01.607749 1021 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.238.252:39755: connect: Connection refused (error 111)
I20260929 10:59:01.608161 1074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:01.608058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.252:39755 (local address 127.0.0.1:47674)
0929 10:59:01.608100 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0929 10:59:01.608131 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.252:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260929 10:59:01.608309 1072 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.238.252:39755: Network error: Client connection negotiation failed: client connection to 127.10.238.252:39755: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.252.keytab.
Entry for principal kudu/127.10.238.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.252.keytab.
I20260929 10:59:01.623557 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.252:39755
--webserver_interface=127.10.238.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.252.keytab
--principal=kudu/127.10.238.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:39561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:33259,127.10.238.253:33389,127.10.238.252:39755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:59:01.657543 1072 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.238.252" port: 39755 } attempt: 1
I20260929 10:59:01.658066 1074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:01.657915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.252:39755 (local address 127.0.0.1:47688)
0929 10:59:01.657983 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0929 10:59:01.658032 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.252:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260929 10:59:01.658409 1074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:01.658305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.252:39755 (local address 127.0.0.1:47704)
0929 10:59:01.658351 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 10:59:01.658382 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.252:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260929 10:59:01.658497 1072 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.238.252:39755: Network error: Client connection negotiation failed: client connection to 127.10.238.252:39755: connect: Connection refused (error 111)
I20260929 10:59:01.665942 997 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.238.253" port: 33389 } attempt: 3
Sep 29 10:59:01 dist-test-slave-8nrt krb5kdc[916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679541, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for kudu/127.10.238.253@KRBTEST.COM
I20260929 10:59:01.673053 1076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:01.666381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54050 (local address 127.10.238.253:33389)
0929 10:59:01.666439 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:01.666442 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:01.666455 (+ 13us) server_negotiation.cc:407] Connection header received
0929 10:59:01.666498 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:01.666503 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:01.666514 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:01.666571 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:01.666922 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:01.667479 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:01.668192 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:01.668419 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:01.668492 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:01.670477 (+ 1985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:59:01.670479 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:59:01.670480 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:59:01.670488 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:59:01.671850 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:01.672213 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:01.672214 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:01.672214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:01.672291 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:01.672482 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:01.672483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:01.672484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:01.672642 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:59:01.672717 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 10:59:01.672881 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 10:59:01.672955 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 10:59:01.673054 999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:01.666297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.253:33389 (local address 127.0.0.1:54050)
0929 10:59:01.666353 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0929 10:59:01.666371 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 10:59:01.666430 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:59:01.666576 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:59:01.666577 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:59:01.666771 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:59:01.666838 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:59:01.666840 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:59:01.667558 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:59:01.667559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:59:01.668100 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:59:01.668102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:59:01.668503 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:59:01.668504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:59:01.668580 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:01.669103 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:59:01.669109 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:59:01.670389 (+ 1280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:59:01.671960 (+ 1571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:59:01.671961 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:59:01.671962 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:59:01.672138 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:59:01.672361 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:59:01.672362 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:59:01.672362 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:59:01.672421 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:59:01.672723 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:59:01.672725 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:59:01.672771 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 10:59:01.672843 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 10:59:01.672954 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260929 10:59:01.673593 997 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.238.252" port: 39755 } has no permanent_uuid. Determining permanent_uuid...
I20260929 10:59:01.673940 999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:01.673856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.252:39755 (local address 127.0.0.1:47716)
0929 10:59:01.673895 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0929 10:59:01.673923 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.252:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260929 10:59:01.674245 999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:01.674161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.252:39755 (local address 127.0.0.1:47730)
0929 10:59:01.674200 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0929 10:59:01.674224 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.252:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260929 10:59:01.674328 997 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.238.252:39755: Network error: Client connection negotiation failed: client connection to 127.10.238.252:39755: connect: Connection refused (error 111)
I20260929 10:59:01.705464 997 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.238.252" port: 39755 } attempt: 1
I20260929 10:59:01.705891 999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:01.705760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.252:39755 (local address 127.0.0.1:47734)
0929 10:59:01.705817 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0929 10:59:01.705852 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.252:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 10:59:01.706213 999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:01.706114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.252:39755 (local address 127.0.0.1:47748)
0929 10:59:01.706170 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0929 10:59:01.706197 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.252:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260929 10:59:01.706103 1079 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 10:59:01.706317 997 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.238.252:39755: Network error: Client connection negotiation failed: client connection to 127.10.238.252:39755: connect: Connection refused (error 111)
W20260929 10:59:01.706332 1079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:59:01.706378 1079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:59:01.706420 1079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:59:01.707595 1079 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:59:01.707656 1079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:59:01.707680 1079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:59:01.707703 1079 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:59:01.707734 1079 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:59:01.709196 1079 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:39561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:33259,127.10.238.253:33389,127.10.238.252:39755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.252.keytab
--principal=kudu/127.10.238.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.252:39755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.238.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:59:01.709482 1079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:59:01.709702 1079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:59:01.714381 1084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:59:01.714493 1079 server_base.cc:1034] running on GCE node
W20260929 10:59:01.714435 1087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:59:01.714406 1085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:59:01.714807 1079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:59:01.715088 1079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:59:01.716246 1079 hybrid_clock.cc:648] HybridClock initialized: now 1790679541716230 us; error 31 us; skew 500 ppm
Sep 29 10:59:01 dist-test-slave-8nrt krb5kdc[916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679541, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:59:01.718782 1079 init.cc:377] Logged in from keytab as kudu/127.10.238.252@KRBTEST.COM (short username kudu)
I20260929 10:59:01.719244 1079 webserver.cc:460] Webserver started at http://127.10.238.252:33201/ using document root <none> and password file <none>
I20260929 10:59:01.719465 1079 fs_manager.cc:359] Metadata directory not provided
I20260929 10:59:01.719529 1079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:59:01.719671 1079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:59:01.720714 1079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/minicluster-data/master-2/data/instance:
uuid: "27cb6fe8a05e4877860f7bbbb801a512"
format_stamp: "Formatted at 2026-09-29 10:59:01 on dist-test-slave-8nrt"
server_key: "8dd2b9cafdc43c54a013fcde25e03c5a"
server_key_iv: "1b4519fba4425fc11c426a1dfcf6dae8"
server_key_version: "encryptionkey@0"
I20260929 10:59:01.721076 1079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/minicluster-data/master-2/wal/instance:
uuid: "27cb6fe8a05e4877860f7bbbb801a512"
format_stamp: "Formatted at 2026-09-29 10:59:01 on dist-test-slave-8nrt"
server_key: "8dd2b9cafdc43c54a013fcde25e03c5a"
server_key_iv: "1b4519fba4425fc11c426a1dfcf6dae8"
server_key_version: "encryptionkey@0"
I20260929 10:59:01.722440 1079 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 10:59:01.723168 1094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:59:01.723330 1079 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 10:59:01.723424 1079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/minicluster-data/master-2/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/minicluster-data/master-2/wal
uuid: "27cb6fe8a05e4877860f7bbbb801a512"
format_stamp: "Formatted at 2026-09-29 10:59:01 on dist-test-slave-8nrt"
server_key: "8dd2b9cafdc43c54a013fcde25e03c5a"
server_key_iv: "1b4519fba4425fc11c426a1dfcf6dae8"
server_key_version: "encryptionkey@0"
I20260929 10:59:01.723505 1079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:59:01.726620 1072 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.238.252" port: 39755 } attempt: 2
I20260929 10:59:01.727002 1074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:01.726896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.252:39755 (local address 127.0.0.1:47756)
0929 10:59:01.726937 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0929 10:59:01.726975 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.252:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260929 10:59:01.727344 1074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:01.727266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.252:39755 (local address 127.0.0.1:47764)
0929 10:59:01.727303 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0929 10:59:01.727327 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.252:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260929 10:59:01.727437 1072 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.238.252:39755: Network error: Client connection negotiation failed: client connection to 127.10.238.252:39755: connect: Connection refused (error 111)
I20260929 10:59:01.741891 1079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:59:01.744172 1079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:59:01.744339 1079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:59:01.748510 1079 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.252:39755
I20260929 10:59:01.748548 1146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.252:39755 every 8 connection(s)
I20260929 10:59:01.748908 1079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/minicluster-data/master-2/data/info.pb
I20260929 10:59:01.749864 1147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:59:01.751468 1147 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.238.254" port: 33259 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 10:59:01 dist-test-slave-8nrt krb5kdc[916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679541, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.252@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:59:01.758426 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 1079
I20260929 10:59:01.758533 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790679274510869-11195-0/minicluster-data/master-2/wal/instance
I20260929 10:59:01.758725 11195 external_mini_cluster.cc:1428] Setting key a7f893e0d7ee167e8a39d6f40fca1670
I20260929 10:59:01.760500 1000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:01.752191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50542 (local address 127.10.238.254:33259)
0929 10:59:01.752238 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:01.752240 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:01.752463 (+ 223us) server_negotiation.cc:407] Connection header received
0929 10:59:01.752987 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:01.752988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:01.753000 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:01.753043 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:01.753733 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:01.754210 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:01.755184 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:01.755379 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:01.755462 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:01.757590 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:59:01.757592 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:59:01.757592 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:59:01.757600 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:59:01.759039 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:01.759553 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:01.759554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:01.759554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:01.759626 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:01.759869 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:01.759870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:01.759871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:01.760027 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:59:01.760100 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 10:59:01.760372 (+ 272us) server_negotiation.cc:299] Negotiation successful
0929 10:59:01.760436 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 10:59:01.760790 1149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:01.752164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:33259 (local address 127.0.0.1:50542)
0929 10:59:01.752346 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 10:59:01.752382 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 10:59:01.752874 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:59:01.753122 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:59:01.753126 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:59:01.753449 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:59:01.753642 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:59:01.753646 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:59:01.754315 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:59:01.754316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:59:01.755087 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:59:01.755090 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:59:01.755469 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:59:01.755470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:59:01.755600 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:01.756094 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:59:01.756102 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:59:01.757499 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:59:01.759139 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:59:01.759143 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:59:01.759145 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:59:01.759463 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:59:01.759708 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:59:01.759708 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:59:01.759709 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:59:01.759805 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:59:01.760096 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:59:01.760100 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:59:01.760215 (+ 115us) client_negotiation.cc:769] Sending connection context
0929 10:59:01.760344 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 10:59:01.760496 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":84,"threads_started":1}
I20260929 10:59:01.761389 1147 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.238.253" port: 33389 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 10:59:01 dist-test-slave-8nrt krb5kdc[916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.238.252@KRBTEST.COM
Sep 29 10:59:01 dist-test-slave-8nrt krb5kdc[916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679541, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.252@KRBTEST.COM for kudu/127.10.238.253@KRBTEST.COM
I20260929 10:59:01.768373 1151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:01.759123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47774 (local address 127.10.238.252:39755)
0929 10:59:01.759314 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:01.759319 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:01.759372 (+ 53us) server_negotiation.cc:407] Connection header received
0929 10:59:01.759418 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:01.759421 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:01.759436 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:01.759517 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:01.760261 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:01.761250 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:01.762125 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:01.762472 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:01.762523 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:01.765198 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:59:01.765205 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:59:01.765205 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:59:01.765211 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:59:01.767110 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:01.767507 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:01.767510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:01.767510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:01.767564 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:01.767755 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:01.767755 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:01.767756 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:01.767934 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:59:01.768052 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 10:59:01.768170 (+ 118us) server_negotiation.cc:299] Negotiation successful
0929 10:59:01.768249 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":103,"threads_started":1}
I20260929 10:59:01.768445 1076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:01.761745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54064 (local address 127.10.238.253:33389)
0929 10:59:01.761813 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:01.761815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:01.761824 (+ 9us) server_negotiation.cc:407] Connection header received
0929 10:59:01.761847 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:01.761848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:01.761858 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:01.761918 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:01.762196 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:01.762730 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:01.763491 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:01.763840 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:01.763930 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:01.765802 (+ 1872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:59:01.765803 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:59:01.765804 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:59:01.765812 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:59:01.767270 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:01.767612 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:01.767614 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:01.767614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:01.767676 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:01.767900 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:01.767901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:01.767901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:01.768030 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:59:01.768101 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 10:59:01.768250 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 10:59:01.768329 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 10:59:01.768424 1149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:01.761634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.253:33389 (local address 127.0.0.1:54064)
0929 10:59:01.761702 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0929 10:59:01.761712 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 10:59:01.761749 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:59:01.761935 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:59:01.761935 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:59:01.762061 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:59:01.762124 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:59:01.762125 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:59:01.762807 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:59:01.762808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:59:01.763390 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:59:01.763392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:59:01.763925 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:59:01.763925 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:59:01.763977 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:01.764349 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:59:01.764354 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:59:01.765709 (+ 1355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:59:01.767347 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:59:01.767348 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:59:01.767348 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:59:01.767527 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:59:01.767758 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:59:01.767759 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:59:01.767759 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:59:01.767840 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:59:01.768098 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:59:01.768098 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:59:01.768140 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 10:59:01.768211 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 10:59:01.768293 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":8}
I20260929 10:59:01.769061 1147 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.238.252" port: 39755 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 10:59:01 dist-test-slave-8nrt krb5kdc[916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679541, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.252@KRBTEST.COM for kudu/127.10.238.252@KRBTEST.COM
Sep 29 10:59:01 dist-test-slave-8nrt krb5kdc[916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:59:01.775801 1149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:01.769334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.252:39755 (local address 127.0.0.1:47788)
0929 10:59:01.769378 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0929 10:59:01.769387 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 10:59:01.769422 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:59:01.769630 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:59:01.769631 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:59:01.769759 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:59:01.769782 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:59:01.769783 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:59:01.770362 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:59:01.770363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:59:01.770815 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:59:01.770816 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:59:01.771183 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:59:01.771184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:59:01.771232 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:01.771613 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:59:01.771617 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:59:01.773463 (+ 1846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:59:01.774817 (+ 1354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:59:01.774818 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:59:01.774818 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:59:01.775015 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:59:01.775192 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:59:01.775193 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:59:01.775193 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:59:01.775256 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:59:01.775469 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:59:01.775470 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:59:01.775506 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 10:59:01.775571 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 10:59:01.775668 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260929 10:59:01.775801 1151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:01.769382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47788 (local address 127.10.238.252:39755)
0929 10:59:01.769543 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:01.769544 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:01.769549 (+ 5us) server_negotiation.cc:407] Connection header received
0929 10:59:01.769562 (+ 13us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:01.769563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:01.769569 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:01.769627 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:01.769847 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:01.770291 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:01.770901 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:01.771101 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:01.771183 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:01.773544 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:59:01.773545 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:59:01.773546 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:59:01.773551 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:59:01.774740 (+ 1189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:01.775085 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:01.775085 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:01.775086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:01.775131 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:01.775317 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:01.775317 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:01.775317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:01.775406 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:59:01.775476 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 10:59:01.775594 (+ 118us) server_negotiation.cc:299] Negotiation successful
0929 10:59:01.775715 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":142}
I20260929 10:59:01.776618 1000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:01.769376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50546 (local address 127.10.238.254:33259)
0929 10:59:01.769412 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:01.769414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:01.769456 (+ 42us) server_negotiation.cc:407] Connection header received
0929 10:59:01.769521 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:01.769522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:01.769530 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:01.769590 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:01.770060 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:01.770506 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:01.771148 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:01.771357 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:01.771437 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:01.773605 (+ 2168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:59:01.773606 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:59:01.773606 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:59:01.773613 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:59:01.775353 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:01.775666 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:01.775668 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:01.775668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:01.775720 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:01.775932 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:01.775933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:01.775934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:01.776079 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:59:01.776246 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 10:59:01.776478 (+ 232us) server_negotiation.cc:299] Negotiation successful
0929 10:59:01.776541 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 10:59:01.777632 1147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27cb6fe8a05e4877860f7bbbb801a512: Bootstrap starting.
I20260929 10:59:01.778632 1147 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27cb6fe8a05e4877860f7bbbb801a512: Neither blocks nor log segments found. Creating new log.
I20260929 10:59:01.778944 1147 log.cc:824] T 00000000000000000000000000000000 P 27cb6fe8a05e4877860f7bbbb801a512: Log is configured to *not* fsync() on all Append() calls
I20260929 10:59:01.779762 1147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27cb6fe8a05e4877860f7bbbb801a512: No bootstrap required, opened a new log
I20260929 10:59:01.781453 997 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.238.252" port: 39755 } attempt: 2
Sep 29 10:59:01 dist-test-slave-8nrt krb5kdc[916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.238.253@KRBTEST.COM
I20260929 10:59:01.781985 1147 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27cb6fe8a05e4877860f7bbbb801a512 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cec287e97e04da2abd963f533be8cdb" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 33259 } } peers { permanent_uuid: "f751508dffee4676861de0d1074550d7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 33389 } } peers { permanent_uuid: "27cb6fe8a05e4877860f7bbbb801a512" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 39755 } }
I20260929 10:59:01.782161 1147 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27cb6fe8a05e4877860f7bbbb801a512 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27cb6fe8a05e4877860f7bbbb801a512, State: Initialized, Role: FOLLOWER
I20260929 10:59:01.782317 1147 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27cb6fe8a05e4877860f7bbbb801a512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cec287e97e04da2abd963f533be8cdb" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 33259 } } peers { permanent_uuid: "f751508dffee4676861de0d1074550d7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 33389 } } peers { permanent_uuid: "27cb6fe8a05e4877860f7bbbb801a512" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 39755 } }
I20260929 10:59:01.782696 1147 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27cb6fe8a05e4877860f7bbbb801a512 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:59:01.782722 1153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27cb6fe8a05e4877860f7bbbb801a512 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cec287e97e04da2abd963f533be8cdb" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 33259 } } peers { permanent_uuid: "f751508dffee4676861de0d1074550d7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 33389 } } peers { permanent_uuid: "27cb6fe8a05e4877860f7bbbb801a512" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 39755 } } }
I20260929 10:59:01.782917 1153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27cb6fe8a05e4877860f7bbbb801a512 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 10:59:01.785039 1164 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 27cb6fe8a05e4877860f7bbbb801a512: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:59:01.785179 1164 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:59:01.785732 1076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:01.777229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54078 (local address 127.10.238.253:33389)
0929 10:59:01.777292 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:01.777294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:01.777301 (+ 7us) server_negotiation.cc:407] Connection header received
0929 10:59:01.777352 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:01.777353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:01.777360 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:01.777403 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:01.777939 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:01.778381 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:01.779304 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:01.779549 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:01.779603 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:01.782134 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:59:01.782135 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:59:01.782136 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:59:01.782142 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:59:01.784060 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:01.784451 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:01.784452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:01.784452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:01.784511 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:01.784728 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:01.784729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:01.784729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:01.784876 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:59:01.785098 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 10:59:01.785513 (+ 415us) server_negotiation.cc:299] Negotiation successful
0929 10:59:01.785604 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Sep 29 10:59:01 dist-test-slave-8nrt krb5kdc[916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679541, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for kudu/127.10.238.252@KRBTEST.COM
I20260929 10:59:01.785125 1079 master_runner.cc:186] Error getting master registration for 127.10.238.254:33259: OK, instance_id { permanent_uuid: "4cec287e97e04da2abd963f533be8cdb" instance_seqno: 1790679541439407 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 10:59:01.788765 1079 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 10:59:01.789558 999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:01.781815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.252:39755 (local address 127.0.0.1:47800)
0929 10:59:01.781866 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0929 10:59:01.781875 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 10:59:01.781914 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:59:01.782206 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:59:01.782207 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:59:01.782364 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:59:01.782403 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:59:01.782406 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:59:01.783242 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:59:01.783244 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:59:01.784073 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:59:01.784077 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:59:01.784623 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:59:01.784624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:59:01.784699 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:01.785129 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:59:01.785134 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:59:01.786667 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:59:01.788302 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:59:01.788302 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:59:01.788303 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:59:01.788466 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:59:01.788655 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:59:01.788656 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:59:01.788656 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:59:01.788717 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:59:01.789162 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:59:01.789163 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:59:01.789222 (+ 59us) client_negotiation.cc:769] Sending connection context
0929 10:59:01.789335 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 10:59:01.789434 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 10:59:01.789551 1151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:01.781999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47800 (local address 127.10.238.252:39755)
0929 10:59:01.782058 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:01.782060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:01.782069 (+ 9us) server_negotiation.cc:407] Connection header received
0929 10:59:01.782087 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:01.782088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:01.782097 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:01.782157 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:01.782492 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:01.783145 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:01.784205 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:01.784497 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:01.784556 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:01.786767 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:59:01.786768 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:59:01.786768 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:59:01.786774 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:59:01.788208 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:01.788532 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:01.788534 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:01.788534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:01.788586 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:01.788799 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:01.788800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:01.788801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:01.788953 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:59:01.789122 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 10:59:01.789367 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 10:59:01.789450 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 10:59:01.790280 11195 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:59:01.791842 997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cec287e97e04da2abd963f533be8cdb: Bootstrap starting.
I20260929 10:59:01.791235 971 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4cec287e97e04da2abd963f533be8cdb"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50546
I20260929 10:59:01.792649 997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4cec287e97e04da2abd963f533be8cdb: Neither blocks nor log segments found. Creating new log.
I20260929 10:59:01.792968 997 log.cc:824] T 00000000000000000000000000000000 P 4cec287e97e04da2abd963f533be8cdb: Log is configured to *not* fsync() on all Append() calls
I20260929 10:59:01.793816 997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cec287e97e04da2abd963f533be8cdb: No bootstrap required, opened a new log
I20260929 10:59:01.795184 997 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4cec287e97e04da2abd963f533be8cdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cec287e97e04da2abd963f533be8cdb" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 33259 } } peers { permanent_uuid: "f751508dffee4676861de0d1074550d7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 33389 } } peers { permanent_uuid: "27cb6fe8a05e4877860f7bbbb801a512" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 39755 } }
I20260929 10:59:01.795327 997 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4cec287e97e04da2abd963f533be8cdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cec287e97e04da2abd963f533be8cdb, State: Initialized, Role: FOLLOWER
I20260929 10:59:01.795414 997 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4cec287e97e04da2abd963f533be8cdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cec287e97e04da2abd963f533be8cdb" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 33259 } } peers { permanent_uuid: "f751508dffee4676861de0d1074550d7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 33389 } } peers { permanent_uuid: "27cb6fe8a05e4877860f7bbbb801a512" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 39755 } }
I20260929 10:59:01.795744 997 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4cec287e97e04da2abd963f533be8cdb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:59:01.795727 1167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cec287e97e04da2abd963f533be8cdb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cec287e97e04da2abd963f533be8cdb" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 33259 } } peers { permanent_uuid: "f751508dffee4676861de0d1074550d7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 33389 } } peers { permanent_uuid: "27cb6fe8a05e4877860f7bbbb801a512" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 39755 } } }
I20260929 10:59:01.795787 971 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4cec287e97e04da2abd963f533be8cdb [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 10:59:01.795899 971 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4cec287e97e04da2abd963f533be8cdb [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:59:01.796640 971 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4cec287e97e04da2abd963f533be8cdb [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cec287e97e04da2abd963f533be8cdb" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 33259 } } peers { permanent_uuid: "f751508dffee4676861de0d1074550d7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 33389 } } peers { permanent_uuid: "27cb6fe8a05e4877860f7bbbb801a512" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 39755 } }
I20260929 10:59:01.796911 971 leader_election.cc:288] T 00000000000000000000000000000000 P 4cec287e97e04da2abd963f533be8cdb [CANDIDATE]: Term 1 election: Requested vote from peers f751508dffee4676861de0d1074550d7 (127.10.238.253:33389), 27cb6fe8a05e4877860f7bbbb801a512 (127.10.238.252:39755)
I20260929 10:59:01.797580 1121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4cec287e97e04da2abd963f533be8cdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "27cb6fe8a05e4877860f7bbbb801a512"
I20260929 10:59:01.797698 1121 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27cb6fe8a05e4877860f7bbbb801a512 [term 0 FOLLOWER]: Advancing to term 1
W20260929 10:59:01.797803 1178 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4cec287e97e04da2abd963f533be8cdb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:59:01.797853 1178 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:59:01.798629 1121 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 27cb6fe8a05e4877860f7bbbb801a512 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cec287e97e04da2abd963f533be8cdb in term 1.
I20260929 10:59:01.798856 1167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cec287e97e04da2abd963f533be8cdb [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:59:01.798978 946 leader_election.cc:302] T 00000000000000000000000000000000 P 4cec287e97e04da2abd963f533be8cdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27cb6fe8a05e4877860f7bbbb801a512, 4cec287e97e04da2abd963f533be8cdb; no voters:
I20260929 10:59:01.799144 1167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4cec287e97e04da2abd963f533be8cdb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:59:01.799319 1167 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4cec287e97e04da2abd963f533be8cdb [term 1 LEADER]: Becoming Leader. State: Replica: 4cec287e97e04da2abd963f533be8cdb, State: Running, Role: LEADER
I20260929 10:59:01.799425 1167 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4cec287e97e04da2abd963f533be8cdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cec287e97e04da2abd963f533be8cdb" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 33259 } } peers { permanent_uuid: "f751508dffee4676861de0d1074550d7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 33389 } } peers { permanent_uuid: "27cb6fe8a05e4877860f7bbbb801a512" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 39755 } }
I20260929 10:59:01.799858 1185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cec287e97e04da2abd963f533be8cdb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4cec287e97e04da2abd963f533be8cdb. Latest consensus state: current_term: 1 leader_uuid: "4cec287e97e04da2abd963f533be8cdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cec287e97e04da2abd963f533be8cdb" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 33259 } } peers { permanent_uuid: "f751508dffee4676861de0d1074550d7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 33389 } } peers { permanent_uuid: "27cb6fe8a05e4877860f7bbbb801a512" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 39755 } } }
I20260929 10:59:01.799953 1185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cec287e97e04da2abd963f533be8cdb [sys.catalog]: This master's current role is: LEADER
I20260929 10:59:01.800380 1189 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:59:01.800770 1189 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:59:01.799086 929 master_runner.cc:186] Error getting master registration for 127.10.238.253:33389: OK, instance_id { permanent_uuid: "f751508dffee4676861de0d1074550d7" instance_seqno: 1790679541585872 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 10:59:01.802289 929 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 10:59:01.808734 1076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:01.798496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54094 (local address 127.10.238.253:33389)
0929 10:59:01.798539 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:01.798541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:01.800346 (+ 1805us) server_negotiation.cc:407] Connection header received
0929 10:59:01.800421 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:01.800422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:01.800434 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:01.800484 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:01.801316 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:01.802054 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:01.803204 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:01.803516 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:01.803620 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:01.805170 (+ 1550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:59:01.805171 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:59:01.805172 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:59:01.805180 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:59:01.807137 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:01.807605 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:01.807606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:01.807606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:01.807682 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:01.807881 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:01.807882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:01.807882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:01.808033 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:59:01.808150 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 10:59:01.808543 (+ 393us) server_negotiation.cc:299] Negotiation successful
0929 10:59:01.808633 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 10:59:01.815871 1000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:01.798880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50552 (local address 127.10.238.254:33259)
0929 10:59:01.798913 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:01.798915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:01.803651 (+ 4736us) server_negotiation.cc:407] Connection header received
0929 10:59:01.804000 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:01.804001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:01.804011 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:01.804040 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:01.805715 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:01.806423 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:01.807977 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:01.808278 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:01.808322 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:01.812348 (+ 4026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:59:01.812349 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:59:01.812350 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:59:01.812357 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:59:01.814037 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:01.814715 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:01.814717 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:01.814717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:01.814784 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:01.815119 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:01.815120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:01.815121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:01.815265 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:59:01.815351 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 10:59:01.815679 (+ 328us) server_negotiation.cc:299] Negotiation successful
0929 10:59:01.815767 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 10:59:01.818120 1151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:01.800130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47814 (local address 127.10.238.252:39755)
0929 10:59:01.800174 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:01.800176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:01.808814 (+ 8638us) server_negotiation.cc:407] Connection header received
0929 10:59:01.809212 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:01.809213 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:01.809225 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:01.809289 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:01.809949 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:01.810668 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:01.811654 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:01.811960 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:01.812028 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:01.814937 (+ 2909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:59:01.814939 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:59:01.814939 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:59:01.814947 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:59:01.816638 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:01.817192 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:01.817193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:01.817194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:01.817295 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:01.817488 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:01.817489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:01.817489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:01.817649 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:59:01.817758 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 10:59:01.817935 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 10:59:01.818019 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 10:59:01.818602 1072 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.238.252" port: 39755 } attempt: 3
I20260929 10:59:01.822489 1121 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 27cb6fe8a05e4877860f7bbbb801a512 [term 1 FOLLOWER]: Refusing update from remote peer 4cec287e97e04da2abd963f533be8cdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
Sep 29 10:59:01 dist-test-slave-8nrt krb5kdc[916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679541, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.253@KRBTEST.COM for kudu/127.10.238.252@KRBTEST.COM
I20260929 10:59:01.824155 1167 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4cec287e97e04da2abd963f533be8cdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "27cb6fe8a05e4877860f7bbbb801a512" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 39755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:59:01.826314 1153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27cb6fe8a05e4877860f7bbbb801a512 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4cec287e97e04da2abd963f533be8cdb. Latest consensus state: current_term: 1 leader_uuid: "4cec287e97e04da2abd963f533be8cdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cec287e97e04da2abd963f533be8cdb" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 33259 } } peers { permanent_uuid: "f751508dffee4676861de0d1074550d7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 33389 } } peers { permanent_uuid: "27cb6fe8a05e4877860f7bbbb801a512" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 39755 } } }
I20260929 10:59:01.826407 1153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27cb6fe8a05e4877860f7bbbb801a512 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:59:01.828329 1185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cec287e97e04da2abd963f533be8cdb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4cec287e97e04da2abd963f533be8cdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cec287e97e04da2abd963f533be8cdb" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 33259 } } peers { permanent_uuid: "f751508dffee4676861de0d1074550d7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 33389 } } peers { permanent_uuid: "27cb6fe8a05e4877860f7bbbb801a512" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 39755 } } }
I20260929 10:59:01.828420 1185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cec287e97e04da2abd963f533be8cdb [sys.catalog]: This master's current role is: LEADER
I20260929 10:59:01.828701 1074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:01.818858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.252:39755 (local address 127.0.0.1:47816)
0929 10:59:01.818906 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0929 10:59:01.818916 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 10:59:01.818949 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:59:01.819109 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:59:01.819110 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:59:01.819280 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:59:01.819326 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:59:01.819328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:59:01.820215 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:59:01.820216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:59:01.821031 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:59:01.821033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:59:01.821719 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:59:01.821720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:59:01.821800 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:01.822504 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:59:01.822521 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:59:01.824069 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:59:01.827200 (+ 3131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:59:01.827201 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:59:01.827202 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:59:01.827451 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:59:01.827779 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:59:01.827780 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:59:01.827780 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:59:01.827855 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:59:01.828300 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:59:01.828301 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:59:01.828361 (+ 60us) client_negotiation.cc:769] Sending connection context
0929 10:59:01.828471 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 10:59:01.828599 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 10:59:01.828970 1153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27cb6fe8a05e4877860f7bbbb801a512 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4cec287e97e04da2abd963f533be8cdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cec287e97e04da2abd963f533be8cdb" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 33259 } } peers { permanent_uuid: "f751508dffee4676861de0d1074550d7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 33389 } } peers { permanent_uuid: "27cb6fe8a05e4877860f7bbbb801a512" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 39755 } } }
I20260929 10:59:01.829062 1153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27cb6fe8a05e4877860f7bbbb801a512 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:59:01.829377 1151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:01.818918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47816 (local address 127.10.238.252:39755)
0929 10:59:01.818972 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:01.818973 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:01.818985 (+ 12us) server_negotiation.cc:407] Connection header received
0929 10:59:01.819017 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:01.819017 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:01.819027 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:01.819102 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:01.819409 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:01.820112 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:01.821129 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:01.821600 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:01.821717 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:01.824185 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:59:01.824186 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:59:01.824186 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:59:01.824193 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:59:01.827030 (+ 2837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:01.827591 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:01.827592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:01.827592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:01.827678 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:01.827951 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:01.827951 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:01.827952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:01.828154 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:59:01.828961 (+ 807us) server_negotiation.cc:1035] Waiting for connection context
0929 10:59:01.829032 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 10:59:01.829204 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 10:59:01.830581 1191 mvcc.cc:205] Tried to move back new op lower bound from 7334623403301388288 to 7334623403211223040. Current Snapshot: MvccSnapshot[applied={T|T < 7334623403301388288}]
I20260929 10:59:01.831586 1189 catalog_manager.cc:1347] Generated new cluster ID: f17470c7419a4feea0b00604e685911f
I20260929 10:59:01.831632 1189 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:59:01.831758 1072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f751508dffee4676861de0d1074550d7: Bootstrap starting.
I20260929 10:59:01.832485 1072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f751508dffee4676861de0d1074550d7: Neither blocks nor log segments found. Creating new log.
I20260929 10:59:01.832798 1072 log.cc:824] T 00000000000000000000000000000000 P f751508dffee4676861de0d1074550d7: Log is configured to *not* fsync() on all Append() calls
I20260929 10:59:01.837961 1072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f751508dffee4676861de0d1074550d7: No bootstrap required, opened a new log
I20260929 10:59:01.840135 1072 raft_consensus.cc:348] T 00000000000000000000000000000000 P f751508dffee4676861de0d1074550d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cec287e97e04da2abd963f533be8cdb" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 33259 } } peers { permanent_uuid: "f751508dffee4676861de0d1074550d7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 33389 } } peers { permanent_uuid: "27cb6fe8a05e4877860f7bbbb801a512" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 39755 } }
I20260929 10:59:01.840271 1072 raft_consensus.cc:729] T 00000000000000000000000000000000 P f751508dffee4676861de0d1074550d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f751508dffee4676861de0d1074550d7, State: Initialized, Role: FOLLOWER
I20260929 10:59:01.840346 1072 consensus_queue.cc:260] T 00000000000000000000000000000000 P f751508dffee4676861de0d1074550d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cec287e97e04da2abd963f533be8cdb" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 33259 } } peers { permanent_uuid: "f751508dffee4676861de0d1074550d7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 33389 } } peers { permanent_uuid: "27cb6fe8a05e4877860f7bbbb801a512" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 39755 } }
I20260929 10:59:01.840668 1072 sys_catalog.cc:564] T 00000000000000000000000000000000 P f751508dffee4676861de0d1074550d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:59:01.842104 1045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f751508dffee4676861de0d1074550d7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:59:01.843053 1045 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f751508dffee4676861de0d1074550d7 [term 1 FOLLOWER]: Refusing update from remote peer 4cec287e97e04da2abd963f533be8cdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:59:01.843266 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4cec287e97e04da2abd963f533be8cdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f751508dffee4676861de0d1074550d7"
I20260929 10:59:01.843335 1185 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4cec287e97e04da2abd963f533be8cdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f751508dffee4676861de0d1074550d7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 33389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:59:01.843827 1189 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:59:01.844115 1046 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f751508dffee4676861de0d1074550d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cec287e97e04da2abd963f533be8cdb in term 1.
I20260929 10:59:01.844763 1189 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:59:01.847947 1196 sys_catalog.cc:455] T 00000000000000000000000000000000 P f751508dffee4676861de0d1074550d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4cec287e97e04da2abd963f533be8cdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cec287e97e04da2abd963f533be8cdb" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 33259 } } peers { permanent_uuid: "f751508dffee4676861de0d1074550d7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 33389 } } peers { permanent_uuid: "27cb6fe8a05e4877860f7bbbb801a512" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 39755 } } }
I20260929 10:59:01.848631 1196 sys_catalog.cc:458] T 00000000000000000000000000000000 P f751508dffee4676861de0d1074550d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:59:01.848703 1196 sys_catalog.cc:455] T 00000000000000000000000000000000 P f751508dffee4676861de0d1074550d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4cec287e97e04da2abd963f533be8cdb. Latest consensus state: current_term: 1 leader_uuid: "4cec287e97e04da2abd963f533be8cdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cec287e97e04da2abd963f533be8cdb" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 33259 } } peers { permanent_uuid: "f751508dffee4676861de0d1074550d7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 33389 } } peers { permanent_uuid: "27cb6fe8a05e4877860f7bbbb801a512" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 39755 } } }
I20260929 10:59:01.848767 1196 sys_catalog.cc:458] T 00000000000000000000000000000000 P f751508dffee4676861de0d1074550d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:59:01.848798 1196 sys_catalog.cc:455] T 00000000000000000000000000000000 P f751508dffee4676861de0d1074550d7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4cec287e97e04da2abd963f533be8cdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cec287e97e04da2abd963f533be8cdb" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 33259 } } peers { permanent_uuid: "f751508dffee4676861de0d1074550d7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 33389 } } peers { permanent_uuid: "27cb6fe8a05e4877860f7bbbb801a512" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 39755 } } }
I20260929 10:59:01.848850 1196 sys_catalog.cc:458] T 00000000000000000000000000000000 P f751508dffee4676861de0d1074550d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:59:01.852919 1189 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4cec287e97e04da2abd963f533be8cdb: Generated new TSK 0
I20260929 10:59:01.853366 1189 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:59:01.854193 1167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cec287e97e04da2abd963f533be8cdb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4cec287e97e04da2abd963f533be8cdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cec287e97e04da2abd963f533be8cdb" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 33259 } } peers { permanent_uuid: "f751508dffee4676861de0d1074550d7" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 33389 } } peers { permanent_uuid: "27cb6fe8a05e4877860f7bbbb801a512" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 39755 } } }
I20260929 10:59:01.854282 1167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cec287e97e04da2abd963f533be8cdb [sys.catalog]: This master's current role is: LEADER
I20260929 10:59:01.854601 1201 mvcc.cc:205] Tried to move back new op lower bound from 7334623403383095296 to 7334623403211223040. Current Snapshot: MvccSnapshot[applied={T|T < 7334623403301388288}]
W20260929 10:59:01.855484 1211 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f751508dffee4676861de0d1074550d7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:59:01.855588 1211 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260929 10:59:01.875409 1076 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:59:01.873798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54100 (local address 127.10.238.253:33389)
0929 10:59:01.873842 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:01.873844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:01.874714 (+ 870us) server_negotiation.cc:407] Connection header received
0929 10:59:01.874820 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:01.874821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:01.874832 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:01.874863 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:01.875318 (+ 455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54100: BlockingRecv error: recv got EOF from 127.0.0.1:54100 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260929 10:59:01.876137 1000 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:59:01.874172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50554 (local address 127.10.238.254:33259)
0929 10:59:01.874212 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:01.874213 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:01.875655 (+ 1442us) server_negotiation.cc:407] Connection header received
0929 10:59:01.875753 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:01.875754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:01.875764 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:01.875792 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:01.876077 (+ 285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50554: BlockingRecv error: recv got EOF from 127.0.0.1:50554 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260929 10:59:01.877007 1151 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:59:01.874453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47826 (local address 127.10.238.252:39755)
0929 10:59:01.874489 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:01.874491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:01.876364 (+ 1873us) server_negotiation.cc:407] Connection header received
0929 10:59:01.876529 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:01.876530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:01.876540 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:01.876570 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:01.876945 (+ 375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47826: BlockingRecv error: recv got EOF from 127.0.0.1:47826 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260929 10:59:01.879386 1076 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:59:01.878650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54106 (local address 127.10.238.253:33389)
0929 10:59:01.878684 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:01.878685 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:01.878886 (+ 201us) server_negotiation.cc:407] Connection header received
0929 10:59:01.878986 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:01.878987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:01.879003 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:01.879032 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:01.879343 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54106: BlockingRecv error: recv got EOF from 127.0.0.1:54106 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260929 10:59:01.882295 1076 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:59:01.881541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54110 (local address 127.10.238.253:33389)
0929 10:59:01.881572 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:01.881574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:01.881796 (+ 222us) server_negotiation.cc:407] Connection header received
0929 10:59:01.881905 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:01.881907 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:01.881916 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:01.881943 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:01.882252 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54110: BlockingRecv error: recv got EOF from 127.0.0.1:54110 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260929 10:59:01.886116 1076 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:59:01.885406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54116 (local address 127.10.238.253:33389)
0929 10:59:01.885438 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:01.885440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:01.885653 (+ 213us) server_negotiation.cc:407] Connection header received
0929 10:59:01.885747 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:01.885748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:01.885758 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:01.885786 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:01.886072 (+ 286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54116: BlockingRecv error: recv got EOF from 127.0.0.1:54116 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260929 10:59:01.891991 1076 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:59:01.891256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54118 (local address 127.10.238.253:33389)
0929 10:59:01.891288 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:01.891290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:01.891528 (+ 238us) server_negotiation.cc:407] Connection header received
0929 10:59:01.891618 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:01.891619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:01.891628 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:01.891656 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:01.891950 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54118: BlockingRecv error: recv got EOF from 127.0.0.1:54118 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260929 10:59:01.902136 1076 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:59:01.901193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54132 (local address 127.10.238.253:33389)
0929 10:59:01.901235 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:01.901237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:01.901560 (+ 323us) server_negotiation.cc:407] Connection header received
0929 10:59:01.901668 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:01.901669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:01.901681 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:01.901712 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:01.902082 (+ 370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54132: BlockingRecv error: recv got EOF from 127.0.0.1:54132 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260929 10:59:01.920462 1076 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:59:01.919477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54144 (local address 127.10.238.253:33389)
0929 10:59:01.919537 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:01.919539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:01.919839 (+ 300us) server_negotiation.cc:407] Connection header received
0929 10:59:01.919948 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:01.919949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:01.919961 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:01.919991 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:01.920402 (+ 411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54144: BlockingRecv error: recv got EOF from 127.0.0.1:54144 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260929 10:59:01.954875 1076 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:59:01.953880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54154 (local address 127.10.238.253:33389)
0929 10:59:01.953927 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:01.953929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:01.954248 (+ 319us) server_negotiation.cc:407] Connection header received
0929 10:59:01.954360 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:01.954362 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:01.954375 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:01.954407 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:01.954815 (+ 408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54154: BlockingRecv error: recv got EOF from 127.0.0.1:54154 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260929 10:59:02.021046 1076 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:59:02.020333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54162 (local address 127.10.238.253:33389)
0929 10:59:02.020389 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:02.020391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:02.020472 (+ 81us) server_negotiation.cc:407] Connection header received
0929 10:59:02.020507 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:02.020508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:02.020520 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:02.020620 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:02.020985 (+ 365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54162: BlockingRecv error: recv got EOF from 127.0.0.1:54162 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260929 10:59:02.150991 1076 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:59:02.150185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54178 (local address 127.10.238.253:33389)
0929 10:59:02.150292 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:02.150294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:02.150383 (+ 89us) server_negotiation.cc:407] Connection header received
0929 10:59:02.150482 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:02.150483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:02.150496 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:02.150569 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:02.150922 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54178: BlockingRecv error: recv got EOF from 127.0.0.1:54178 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260929 10:59:02.409030 1076 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:59:02.408288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54180 (local address 127.10.238.253:33389)
0929 10:59:02.408375 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:02.408377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:02.408387 (+ 10us) server_negotiation.cc:407] Connection header received
0929 10:59:02.408485 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:02.408486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:02.408498 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:02.408568 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:02.408959 (+ 391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54180: BlockingRecv error: recv got EOF from 127.0.0.1:54180 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 10:59:02.785708 1164 catalog_manager.cc:1259] Loaded cluster ID: f17470c7419a4feea0b00604e685911f
I20260929 10:59:02.785787 1164 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 27cb6fe8a05e4877860f7bbbb801a512: loading cluster ID for follower catalog manager: success
I20260929 10:59:02.786540 1164 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 27cb6fe8a05e4877860f7bbbb801a512: acquiring CA information for follower catalog manager: success
I20260929 10:59:02.786849 1164 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 27cb6fe8a05e4877860f7bbbb801a512: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 10:59:02.855978 1211 catalog_manager.cc:1259] Loaded cluster ID: f17470c7419a4feea0b00604e685911f
I20260929 10:59:02.856042 1211 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f751508dffee4676861de0d1074550d7: loading cluster ID for follower catalog manager: success
I20260929 10:59:02.856835 1211 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f751508dffee4676861de0d1074550d7: acquiring CA information for follower catalog manager: success
I20260929 10:59:02.857093 1211 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f751508dffee4676861de0d1074550d7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 10:59:02.925971 1280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:02.922173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54190 (local address 127.10.238.253:33389)
0929 10:59:02.922314 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:02.922316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:02.922346 (+ 30us) server_negotiation.cc:407] Connection header received
0929 10:59:02.922442 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:02.922443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:02.922534 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:02.922627 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:59:02.922742 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:02.923593 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:02.924827 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:02.925197 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:02.925302 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:02.925535 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:59:02.925684 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:59:02.925768 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 10:59:02.925828 (+ 60us) server_negotiation.cc:299] Negotiation successful
0929 10:59:02.925857 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":80,"threads_started":1}
I20260929 10:59:02.931363 1286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:02.928433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47838 (local address 127.10.238.252:39755)
0929 10:59:02.928552 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:02.928555 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:02.928564 (+ 9us) server_negotiation.cc:407] Connection header received
0929 10:59:02.928608 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:02.928609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:02.928655 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:02.928720 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:59:02.928883 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:02.929659 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:02.930501 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:02.930725 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:02.930806 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:02.930975 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:59:02.931096 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:59:02.931202 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 10:59:02.931243 (+ 41us) server_negotiation.cc:299] Negotiation successful
0929 10:59:02.931267 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":67,"threads_started":1}
I20260929 10:59:02.936694 1280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:02.933405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54200 (local address 127.10.238.253:33389)
0929 10:59:02.933459 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:02.933461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:02.933539 (+ 78us) server_negotiation.cc:407] Connection header received
0929 10:59:02.933627 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:02.933629 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:02.933639 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:02.933719 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:59:02.933859 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:02.934724 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:02.935876 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:02.936187 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:02.936257 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:02.936405 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:59:02.936442 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:59:02.936493 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 10:59:02.936548 (+ 55us) server_negotiation.cc:299] Negotiation successful
0929 10:59:02.936571 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 10:59:02.936755 1286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:02.933504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47844 (local address 127.10.238.252:39755)
0929 10:59:02.933547 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:02.933549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:02.933725 (+ 176us) server_negotiation.cc:407] Connection header received
0929 10:59:02.933820 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:02.933821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:02.933845 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:02.933890 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:59:02.934076 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:02.934815 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:02.935965 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:02.936265 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:02.936304 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:02.936508 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:59:02.936537 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:59:02.936562 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0929 10:59:02.936661 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 10:59:02.936687 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 10:59:02.938000 1293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:02.933292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50568 (local address 127.10.238.254:33259)
0929 10:59:02.933404 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:02.933407 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:02.933572 (+ 165us) server_negotiation.cc:407] Connection header received
0929 10:59:02.933720 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:02.933721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:02.933774 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:02.933920 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:59:02.934917 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:02.935684 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:02.936718 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:02.937409 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:02.937485 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:02.937675 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:59:02.937764 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:59:02.937834 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 10:59:02.937890 (+ 56us) server_negotiation.cc:299] Negotiation successful
0929 10:59:02.937914 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":62,"threads_started":1}
I20260929 10:59:02.940109 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 929
I20260929 10:59:02.947328 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 1004
I20260929 10:59:02.954596 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 1079
2026-09-29T10:59:02Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2836 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:59:03 dist-test-slave-8nrt krb5kdc[1297](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:59:03 dist-test-slave-8nrt krb5kdc[1297](info): set up 2 sockets
Sep 29 10:59:03 dist-test-slave-8nrt krb5kdc[1297](info): commencing operation
krb5kdc: starting...
W20260929 10:59:03.993858 11195 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/krb5kdc/test-user.keytab.
Sep 29 10:59:04 dist-test-slave-8nrt krb5kdc[1297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:59:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:59:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
Entry for principal kudu/127.10.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab.
I20260929 10:59:04.112442 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:35497
--webserver_interface=127.10.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:46615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:35497,127.10.238.253:46213,127.10.238.252:34939
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:59:04.192569 1310 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 10:59:04.192790 1310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:59:04.192807 1310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:59:04.192827 1310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:59:04.193957 1310 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:59:04.193993 1310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:59:04.194008 1310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:59:04.194020 1310 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:59:04.194034 1310 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:59:04.195418 1310 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:46615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:35497,127.10.238.253:46213,127.10.238.252:34939
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.254.keytab
--principal=kudu/127.10.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.254:35497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:59:04.195611 1310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:59:04.195781 1310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 10:59:04.200606 1310 server_base.cc:1034] running on GCE node
W20260929 10:59:04.200609 1318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:59:04.200578 1315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:59:04.200573 1316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:59:04.201150 1310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:59:04.201483 1310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:59:04.202638 1310 hybrid_clock.cc:648] HybridClock initialized: now 1790679544202620 us; error 49 us; skew 500 ppm
Sep 29 10:59:04 dist-test-slave-8nrt krb5kdc[1297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679544, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:59:04.205029 1310 init.cc:377] Logged in from keytab as kudu/127.10.238.254@KRBTEST.COM (short username kudu)
I20260929 10:59:04.205503 1310 webserver.cc:460] Webserver started at http://127.10.238.254:41655/ using document root <none> and password file <none>
I20260929 10:59:04.205718 1310 fs_manager.cc:359] Metadata directory not provided
I20260929 10:59:04.205782 1310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:59:04.205933 1310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:59:04.206954 1310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/minicluster-data/master-0/data/instance:
uuid: "0b0dce2aa03b4ad49f7d70a2c33f1d2d"
format_stamp: "Formatted at 2026-09-29 10:59:04 on dist-test-slave-8nrt"
server_key: "e484347283aba33685baa435f8630cb5"
server_key_iv: "771c4188c485d26cb73626d395d275c0"
server_key_version: "encryptionkey@0"
I20260929 10:59:04.207317 1310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/minicluster-data/master-0/wal/instance:
uuid: "0b0dce2aa03b4ad49f7d70a2c33f1d2d"
format_stamp: "Formatted at 2026-09-29 10:59:04 on dist-test-slave-8nrt"
server_key: "e484347283aba33685baa435f8630cb5"
server_key_iv: "771c4188c485d26cb73626d395d275c0"
server_key_version: "encryptionkey@0"
I20260929 10:59:04.208603 1310 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 10:59:04.209362 1325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:59:04.209576 1310 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:59:04.209685 1310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/minicluster-data/master-0/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/minicluster-data/master-0/wal
uuid: "0b0dce2aa03b4ad49f7d70a2c33f1d2d"
format_stamp: "Formatted at 2026-09-29 10:59:04 on dist-test-slave-8nrt"
server_key: "e484347283aba33685baa435f8630cb5"
server_key_iv: "771c4188c485d26cb73626d395d275c0"
server_key_version: "encryptionkey@0"
I20260929 10:59:04.209769 1310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:59:04.245116 1310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:59:04.247567 1310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:59:04.247740 1310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:59:04.251907 1310 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.254:35497
I20260929 10:59:04.251953 1377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.254:35497 every 8 connection(s)
I20260929 10:59:04.252310 1310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/minicluster-data/master-0/data/info.pb
I20260929 10:59:04.253201 1378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:59:04.254779 1378 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.238.254" port: 35497 } has no permanent_uuid. Determining permanent_uuid...
I20260929 10:59:04.257123 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 1310
I20260929 10:59:04.257201 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/minicluster-data/master-0/wal/instance
I20260929 10:59:04.257416 11195 external_mini_cluster.cc:1428] Setting key ceae1e58a981891caf908e1fd249269f
Sep 29 10:59:04 dist-test-slave-8nrt krb5kdc[1297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679544, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.238.253@KRBTEST.COM; defaulting to no policy
I20260929 10:59:04.266403 1380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:04.255393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:35497 (local address 127.0.0.1:57612)
0929 10:59:04.255546 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 10:59:04.255604 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 10:59:04.256055 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:59:04.256344 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:59:04.256347 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:59:04.256586 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:59:04.256756 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:59:04.256761 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:59:04.257978 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:59:04.257979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:59:04.258835 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:59:04.258839 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:59:04.259331 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:59:04.259332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:59:04.259416 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:04.261496 (+ 2080us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:59:04.261508 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:59:04.263119 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:59:04.264861 (+ 1742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:59:04.264867 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:59:04.264869 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:59:04.265119 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:59:04.265403 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:59:04.265404 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:59:04.265404 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:59:04.265497 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:59:04.265842 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:59:04.265848 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:59:04.265908 (+ 60us) client_negotiation.cc:769] Sending connection context
0929 10:59:04.266010 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 10:59:04.266233 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":89,"thread_start_us":66,"threads_started":1}
I20260929 10:59:04.266413 1381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:04.255444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57612 (local address 127.10.238.254:35497)
0929 10:59:04.255623 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:04.255628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:04.255787 (+ 159us) server_negotiation.cc:407] Connection header received
0929 10:59:04.256193 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:04.256195 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:04.256223 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:04.256312 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:04.256843 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:04.257865 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:04.258928 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:04.259241 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:04.259325 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:04.263210 (+ 3885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:59:04.263217 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:59:04.263217 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:59:04.263224 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:59:04.264744 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:04.265215 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:04.265217 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:04.265218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:04.265338 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:04.265560 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:04.265562 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:04.265562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:04.265773 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:59:04.265862 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 10:59:04.266042 (+ 180us) server_negotiation.cc:299] Negotiation successful
0929 10:59:04.266170 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":111,"threads_started":1}
I20260929 10:59:04.267292 1378 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.238.253" port: 46213 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.253@KRBTEST.COM" created.
I20260929 10:59:04.267777 1380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:04.267636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.253:46213 (local address 127.0.0.1:49352)
0929 10:59:04.267690 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0929 10:59:04.267742 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.253:46213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 10:59:04.267930 1329 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.238.253:46213: connect: Connection refused (error 111)
I20260929 10:59:04.268342 1380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:04.268248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.253:46213 (local address 127.0.0.1:49358)
0929 10:59:04.268290 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0929 10:59:04.268314 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.253:46213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260929 10:59:04.268435 1378 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.238.253:46213: Network error: Client connection negotiation failed: client connection to 127.10.238.253:46213: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.253.keytab.
Entry for principal kudu/127.10.238.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.253.keytab.
I20260929 10:59:04.281913 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.253:46213
--webserver_interface=127.10.238.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.253.keytab
--principal=kudu/127.10.238.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:46615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:35497,127.10.238.253:46213,127.10.238.252:34939
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:59:04.317648 1378 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.238.253" port: 46213 } attempt: 1
I20260929 10:59:04.318341 1380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:04.318195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.253:46213 (local address 127.0.0.1:49372)
0929 10:59:04.318264 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0929 10:59:04.318309 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.253:46213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 10:59:04.318796 1380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:04.318707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.253:46213 (local address 127.0.0.1:49382)
0929 10:59:04.318753 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 10:59:04.318781 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.253:46213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260929 10:59:04.318899 1378 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.238.253:46213: Network error: Client connection negotiation failed: client connection to 127.10.238.253:46213: connect: Connection refused (error 111)
W20260929 10:59:04.364718 1385 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 10:59:04.364936 1385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:59:04.364956 1385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:59:04.365002 1385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:59:04.366237 1385 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:59:04.366285 1385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:59:04.366302 1385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:59:04.366355 1385 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:59:04.366384 1385 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:59:04.367908 1385 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:46615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:35497,127.10.238.253:46213,127.10.238.252:34939
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.253.keytab
--principal=kudu/127.10.238.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.253:46213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.238.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:59:04.368166 1385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:59:04.368399 1385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:59:04.373485 1391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:59:04.373522 1390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:59:04.373531 1393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:59:04.373576 1385 server_base.cc:1034] running on GCE node
I20260929 10:59:04.373888 1385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:59:04.374171 1385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:59:04.375339 1385 hybrid_clock.cc:648] HybridClock initialized: now 1790679544375323 us; error 39 us; skew 500 ppm
Sep 29 10:59:04 dist-test-slave-8nrt krb5kdc[1297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679544, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:59:04.378048 1385 init.cc:377] Logged in from keytab as kudu/127.10.238.253@KRBTEST.COM (short username kudu)
I20260929 10:59:04.378472 1385 webserver.cc:460] Webserver started at http://127.10.238.253:44029/ using document root <none> and password file <none>
I20260929 10:59:04.378698 1385 fs_manager.cc:359] Metadata directory not provided
I20260929 10:59:04.378764 1385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:59:04.378907 1385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:59:04.379976 1385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/minicluster-data/master-1/data/instance:
uuid: "f05dbc3103cb4f9eb3519b44907b66a6"
format_stamp: "Formatted at 2026-09-29 10:59:04 on dist-test-slave-8nrt"
server_key: "4d481b3834ec427277d17b490e294518"
server_key_iv: "2146b831d4630cddf3433d5c9b2934dd"
server_key_version: "encryptionkey@0"
I20260929 10:59:04.380350 1385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/minicluster-data/master-1/wal/instance:
uuid: "f05dbc3103cb4f9eb3519b44907b66a6"
format_stamp: "Formatted at 2026-09-29 10:59:04 on dist-test-slave-8nrt"
server_key: "4d481b3834ec427277d17b490e294518"
server_key_iv: "2146b831d4630cddf3433d5c9b2934dd"
server_key_version: "encryptionkey@0"
I20260929 10:59:04.381794 1385 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 10:59:04.382536 1400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:59:04.382860 1385 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:59:04.382961 1385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/minicluster-data/master-1/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/minicluster-data/master-1/wal
uuid: "f05dbc3103cb4f9eb3519b44907b66a6"
format_stamp: "Formatted at 2026-09-29 10:59:04 on dist-test-slave-8nrt"
server_key: "4d481b3834ec427277d17b490e294518"
server_key_iv: "2146b831d4630cddf3433d5c9b2934dd"
server_key_version: "encryptionkey@0"
I20260929 10:59:04.383033 1385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:59:04.387048 1378 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.238.253" port: 46213 } attempt: 2
I20260929 10:59:04.387481 1380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:04.387350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.253:46213 (local address 127.0.0.1:49394)
0929 10:59:04.387411 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0929 10:59:04.387451 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.253:46213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 10:59:04.387892 1380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:04.387790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.253:46213 (local address 127.0.0.1:49398)
0929 10:59:04.387832 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0929 10:59:04.387860 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.253:46213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260929 10:59:04.387984 1378 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.238.253:46213: Network error: Client connection negotiation failed: client connection to 127.10.238.253:46213: connect: Connection refused (error 111)
I20260929 10:59:04.421172 1385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:59:04.423578 1385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:59:04.423753 1385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:59:04.427917 1385 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.253:46213
I20260929 10:59:04.427959 1452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.253:46213 every 8 connection(s)
I20260929 10:59:04.428337 1385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/minicluster-data/master-1/data/info.pb
I20260929 10:59:04.429240 1453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:59:04.430805 1453 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.238.254" port: 35497 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 10:59:04 dist-test-slave-8nrt krb5kdc[1297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679544, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.253@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:59:04.436656 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 1385
I20260929 10:59:04.436779 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/minicluster-data/master-1/wal/instance
I20260929 10:59:04.436985 11195 external_mini_cluster.cc:1428] Setting key 676231121ec668585dfb516324036f32
I20260929 10:59:04.440202 1381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:04.431532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57624 (local address 127.10.238.254:35497)
0929 10:59:04.431624 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:04.431626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:04.431786 (+ 160us) server_negotiation.cc:407] Connection header received
0929 10:59:04.432255 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:04.432256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:04.432268 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:04.432332 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:04.432915 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:04.433479 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:04.434423 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:04.434630 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:04.434679 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:04.437074 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:59:04.437076 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:59:04.437077 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:59:04.437096 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:59:04.438817 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:04.439199 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:04.439200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:04.439200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:04.439265 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:04.439485 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:04.439486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:04.439486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:04.439651 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:59:04.439722 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 10:59:04.439961 (+ 239us) server_negotiation.cc:299] Negotiation successful
0929 10:59:04.440057 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 10:59:04.440217 1455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:04.431498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:35497 (local address 127.0.0.1:57624)
0929 10:59:04.431656 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 10:59:04.431699 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 10:59:04.432158 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:59:04.432407 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:59:04.432410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:59:04.432645 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:59:04.432838 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:59:04.432841 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:59:04.433563 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:59:04.433563 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:59:04.434348 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:59:04.434350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:59:04.434706 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:59:04.434707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:59:04.434871 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:04.435401 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:59:04.435410 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:59:04.436941 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:59:04.438899 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:59:04.438906 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:59:04.438908 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:59:04.439125 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:59:04.439338 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:59:04.439339 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:59:04.439339 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:59:04.439437 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:59:04.439717 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:59:04.439722 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:59:04.439836 (+ 114us) client_negotiation.cc:769] Sending connection context
0929 10:59:04.439918 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 10:59:04.440041 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":81,"threads_started":1}
I20260929 10:59:04.440997 1453 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.238.253" port: 46213 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.10.238.252@KRBTEST.COM; defaulting to no policy
Sep 29 10:59:04 dist-test-slave-8nrt krb5kdc[1297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679544, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.253@KRBTEST.COM for kudu/127.10.238.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.238.252@KRBTEST.COM" created.
I20260929 10:59:04.448779 1455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:04.441294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.253:46213 (local address 127.0.0.1:49400)
0929 10:59:04.441328 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0929 10:59:04.441338 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 10:59:04.441377 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:59:04.441634 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:59:04.441636 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:59:04.441802 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:59:04.441871 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:59:04.441873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:59:04.442701 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:59:04.442702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:59:04.443535 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:59:04.443538 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:59:04.443991 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:59:04.443992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:59:04.444074 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:04.444430 (+ 356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:59:04.444435 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:59:04.445840 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:59:04.447520 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:59:04.447521 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:59:04.447521 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:59:04.447704 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:59:04.447995 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:59:04.447996 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:59:04.447996 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:59:04.448061 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:59:04.448387 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:59:04.448388 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:59:04.448436 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 10:59:04.448517 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 10:59:04.448624 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260929 10:59:04.448805 1457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:04.441375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49400 (local address 127.10.238.253:46213)
0929 10:59:04.441501 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:04.441504 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:04.441520 (+ 16us) server_negotiation.cc:407] Connection header received
0929 10:59:04.441534 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:04.441536 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:04.441545 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:04.441601 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:04.441947 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:04.442619 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:04.443619 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:04.443918 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:04.443970 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:04.445959 (+ 1989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:59:04.445967 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:59:04.445968 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:59:04.445974 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:59:04.447447 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:04.447844 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:04.447848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:04.447848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:04.447920 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:04.448125 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:04.448126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:04.448127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:04.448319 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:59:04.448387 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 10:59:04.448541 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 10:59:04.448705 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":104,"thread_start_us":79,"threads_started":1}
I20260929 10:59:04.449415 1453 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.238.252" port: 34939 } has no permanent_uuid. Determining permanent_uuid...
I20260929 10:59:04.449762 1455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:04.449640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.252:34939 (local address 127.0.0.1:34954)
0929 10:59:04.449691 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0929 10:59:04.449737 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.252:34939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260929 10:59:04.449921 1402 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.238.252:34939: connect: Connection refused (error 111)
I20260929 10:59:04.450361 1455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:04.450284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.252:34939 (local address 127.0.0.1:34966)
0929 10:59:04.450323 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0929 10:59:04.450347 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.252:34939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260929 10:59:04.450485 1453 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.238.252:34939: Network error: Client connection negotiation failed: client connection to 127.10.238.252:34939: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.238.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.252.keytab.
Entry for principal kudu/127.10.238.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.252.keytab.
I20260929 10:59:04.463768 11195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc60rDW/build/release/bin/kudu
/tmp/dist-test-taskc60rDW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.252:34939
--webserver_interface=127.10.238.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.252.keytab
--principal=kudu/127.10.238.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.238.212:46615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.238.254:35497,127.10.238.253:46213,127.10.238.252:34939
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:59:04.479123 1378 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.238.253" port: 46213 } attempt: 3
Sep 29 10:59:04 dist-test-slave-8nrt krb5kdc[1297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679544, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for kudu/127.10.238.253@KRBTEST.COM
I20260929 10:59:04.486276 1457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:04.479561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49410 (local address 127.10.238.253:46213)
0929 10:59:04.479661 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:04.479663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:04.479672 (+ 9us) server_negotiation.cc:407] Connection header received
0929 10:59:04.479699 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:04.479700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:04.479754 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:04.479878 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:04.480219 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:04.480737 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:04.481519 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:04.481787 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:04.481873 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:04.483682 (+ 1809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:59:04.483683 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:59:04.483684 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:59:04.483691 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:59:04.485088 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:04.485517 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:04.485518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:04.485519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:04.485575 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:04.485768 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:04.485769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:04.485770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:04.485883 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:59:04.485956 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 10:59:04.486103 (+ 147us) server_negotiation.cc:299] Negotiation successful
0929 10:59:04.486180 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 10:59:04.486284 1380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:04.479473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.253:46213 (local address 127.0.0.1:49410)
0929 10:59:04.479531 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0929 10:59:04.479543 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:59:04.479592 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:59:04.479893 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:59:04.479894 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:59:04.480065 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:59:04.480141 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:59:04.480143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:59:04.480818 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:59:04.480820 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:59:04.481435 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:59:04.481437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:59:04.481872 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:59:04.481874 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:59:04.481955 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:04.482425 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:59:04.482429 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:59:04.483597 (+ 1168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:59:04.485201 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:59:04.485202 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:59:04.485203 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:59:04.485423 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:59:04.485639 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:59:04.485640 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:59:04.485640 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:59:04.485700 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:59:04.485948 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:59:04.485948 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:59:04.485995 (+ 47us) client_negotiation.cc:769] Sending connection context
0929 10:59:04.486079 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 10:59:04.486176 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260929 10:59:04.486698 1378 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.238.252" port: 34939 } has no permanent_uuid. Determining permanent_uuid...
I20260929 10:59:04.487051 1380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:04.486952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.252:34939 (local address 127.0.0.1:34968)
0929 10:59:04.486998 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 10:59:04.487025 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.252:34939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260929 10:59:04.487391 1380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:04.487305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.252:34939 (local address 127.0.0.1:34974)
0929 10:59:04.487346 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0929 10:59:04.487369 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.252:34939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260929 10:59:04.487507 1378 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.238.252:34939: Network error: Client connection negotiation failed: client connection to 127.10.238.252:34939: connect: Connection refused (error 111)
I20260929 10:59:04.499696 1453 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.238.252" port: 34939 } attempt: 1
I20260929 10:59:04.500097 1455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:04.499987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.252:34939 (local address 127.0.0.1:34976)
0929 10:59:04.500041 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0929 10:59:04.500074 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.252:34939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 10:59:04.500408 1455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:04.500321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.252:34939 (local address 127.0.0.1:34980)
0929 10:59:04.500362 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0929 10:59:04.500389 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.252:34939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260929 10:59:04.500502 1453 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.238.252:34939: Network error: Client connection negotiation failed: client connection to 127.10.238.252:34939: connect: Connection refused (error 111)
I20260929 10:59:04.518638 1378 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.238.252" port: 34939 } attempt: 1
I20260929 10:59:04.519085 1380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:04.518958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.252:34939 (local address 127.0.0.1:34996)
0929 10:59:04.519017 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0929 10:59:04.519057 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.252:34939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260929 10:59:04.519412 1380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:04.519318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.252:34939 (local address 127.0.0.1:35010)
0929 10:59:04.519359 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0929 10:59:04.519394 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.252:34939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260929 10:59:04.519495 1378 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.238.252:34939: Network error: Client connection negotiation failed: client connection to 127.10.238.252:34939: connect: Connection refused (error 111)
W20260929 10:59:04.546768 1460 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 10:59:04.547014 1460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:59:04.547043 1460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:59:04.547083 1460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:59:04.548316 1460 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:59:04.548377 1460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:59:04.548404 1460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:59:04.548434 1460 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:59:04.548458 1460 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:59:04.549964 1460 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.238.212:46615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.238.254:35497,127.10.238.253:46213,127.10.238.252:34939
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/krb5kdc/kudu_127.10.238.252.keytab
--principal=kudu/127.10.238.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.238.252:34939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.238.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 10:01:35 UTC on ccycloud.quasar-ehiyrj.root.comops.site
build id 4679902
I20260929 10:59:04.550242 1460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:59:04.550462 1460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:59:04.555140 1465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:59:04.555114 1468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:59:04.555344 1460 server_base.cc:1034] running on GCE node
W20260929 10:59:04.555124 1466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:59:04.555649 1460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:59:04.555928 1460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:59:04.557088 1460 hybrid_clock.cc:648] HybridClock initialized: now 1790679544557072 us; error 44 us; skew 500 ppm
Sep 29 10:59:04 dist-test-slave-8nrt krb5kdc[1297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679544, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:59:04.559546 1460 init.cc:377] Logged in from keytab as kudu/127.10.238.252@KRBTEST.COM (short username kudu)
I20260929 10:59:04.559988 1460 webserver.cc:460] Webserver started at http://127.10.238.252:40291/ using document root <none> and password file <none>
I20260929 10:59:04.560212 1460 fs_manager.cc:359] Metadata directory not provided
I20260929 10:59:04.560277 1460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:59:04.560420 1460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:59:04.561512 1460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/minicluster-data/master-2/data/instance:
uuid: "732d35126b1f471cb64cce86634a9a11"
format_stamp: "Formatted at 2026-09-29 10:59:04 on dist-test-slave-8nrt"
server_key: "a5ce267b37caac745b1eba2d86d66c93"
server_key_iv: "a17cd36514dd9d41a070a03ea2f684bc"
server_key_version: "encryptionkey@0"
I20260929 10:59:04.561864 1460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/minicluster-data/master-2/wal/instance:
uuid: "732d35126b1f471cb64cce86634a9a11"
format_stamp: "Formatted at 2026-09-29 10:59:04 on dist-test-slave-8nrt"
server_key: "a5ce267b37caac745b1eba2d86d66c93"
server_key_iv: "a17cd36514dd9d41a070a03ea2f684bc"
server_key_version: "encryptionkey@0"
I20260929 10:59:04.563143 1460 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 10:59:04.563788 1475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:59:04.563972 1460 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 10:59:04.564064 1460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/minicluster-data/master-2/data,/tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/minicluster-data/master-2/wal
uuid: "732d35126b1f471cb64cce86634a9a11"
format_stamp: "Formatted at 2026-09-29 10:59:04 on dist-test-slave-8nrt"
server_key: "a5ce267b37caac745b1eba2d86d66c93"
server_key_iv: "a17cd36514dd9d41a070a03ea2f684bc"
server_key_version: "encryptionkey@0"
I20260929 10:59:04.564136 1460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:59:04.568629 1453 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.238.252" port: 34939 } attempt: 2
I20260929 10:59:04.569011 1455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:04.568896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.252:34939 (local address 127.0.0.1:35024)
0929 10:59:04.568951 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0929 10:59:04.568985 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.252:34939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 10:59:04.569375 1455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:04.569296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.252:34939 (local address 127.0.0.1:35032)
0929 10:59:04.569326 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0929 10:59:04.569356 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.238.252:34939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260929 10:59:04.569450 1453 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.238.252:34939: Network error: Client connection negotiation failed: client connection to 127.10.238.252:34939: connect: Connection refused (error 111)
I20260929 10:59:04.575073 1460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:59:04.577318 1460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:59:04.577466 1460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:59:04.581466 1460 rpc_server.cc:307] RPC server started. Bound to: 127.10.238.252:34939
I20260929 10:59:04.581508 1527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.238.252:34939 every 8 connection(s)
I20260929 10:59:04.581861 1460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/minicluster-data/master-2/data/info.pb
I20260929 10:59:04.582681 1528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:59:04.584213 1528 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.238.254" port: 35497 } has no permanent_uuid. Determining permanent_uuid...
I20260929 10:59:04.588675 11195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc60rDW/build/release/bin/kudu as pid 1460
I20260929 10:59:04.588771 11195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc60rDW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790679274510869-11195-0/minicluster-data/master-2/wal/instance
I20260929 10:59:04.589010 11195 external_mini_cluster.cc:1428] Setting key 8fe40c511de0865e71349007acfc46b9
Sep 29 10:59:04 dist-test-slave-8nrt krb5kdc[1297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679544, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.252@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:59:04.593200 1381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:04.584869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57628 (local address 127.10.238.254:35497)
0929 10:59:04.584947 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:04.584949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:04.585112 (+ 163us) server_negotiation.cc:407] Connection header received
0929 10:59:04.585650 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:04.585651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:04.585661 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:04.585712 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:04.586316 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:04.586796 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:04.587737 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:04.587926 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:04.587998 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:04.590244 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:59:04.590246 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:59:04.590246 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:59:04.590254 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:59:04.591657 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:04.592108 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:04.592110 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:04.592110 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:04.592164 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:04.592440 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:04.592441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:04.592442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:04.592594 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:59:04.592668 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 10:59:04.593017 (+ 349us) server_negotiation.cc:299] Negotiation successful
0929 10:59:04.593106 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 10:59:04.593367 1530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:04.584832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.254:35497 (local address 127.0.0.1:57628)
0929 10:59:04.584999 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 10:59:04.585033 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 10:59:04.585563 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:59:04.585765 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:59:04.585768 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:59:04.586040 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:59:04.586234 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:59:04.586238 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:59:04.586875 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:59:04.586876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:59:04.587646 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:59:04.587647 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:59:04.588010 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:59:04.588011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:59:04.588107 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:04.588612 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:59:04.588619 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:59:04.590163 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:59:04.591741 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:59:04.591748 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:59:04.591751 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:59:04.592026 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:59:04.592230 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:59:04.592231 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:59:04.592231 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:59:04.592367 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:59:04.592692 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:59:04.592705 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:59:04.592864 (+ 159us) client_negotiation.cc:769] Sending connection context
0929 10:59:04.592981 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 10:59:04.593112 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":89,"threads_started":1}
I20260929 10:59:04.593926 1528 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.238.253" port: 46213 } has no permanent_uuid. Determining permanent_uuid...
I20260929 10:59:04.594635 1378 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.238.252" port: 34939 } attempt: 2
Sep 29 10:59:04 dist-test-slave-8nrt krb5kdc[1297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.238.252@KRBTEST.COM
Sep 29 10:59:04 dist-test-slave-8nrt krb5kdc[1297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679544, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.252@KRBTEST.COM for kudu/127.10.238.253@KRBTEST.COM
I20260929 10:59:04.598521 1532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:04.589357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35044 (local address 127.10.238.252:34939)
0929 10:59:04.589534 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:04.589539 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:04.589598 (+ 59us) server_negotiation.cc:407] Connection header received
0929 10:59:04.589663 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:04.589666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:04.589686 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:04.589752 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:04.590434 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:04.591099 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:04.591817 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:04.592302 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:04.592369 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:04.595227 (+ 2858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:59:04.595234 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:59:04.595234 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:59:04.595241 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:59:04.597054 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:04.597482 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:04.597485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:04.597485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:04.597545 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:04.597741 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:04.597741 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:04.597742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:04.597877 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:59:04.598043 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 10:59:04.598293 (+ 250us) server_negotiation.cc:299] Negotiation successful
0929 10:59:04.598399 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":100,"threads_started":1}
Sep 29 10:59:04 dist-test-slave-8nrt krb5kdc[1297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679544, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.254@KRBTEST.COM for kudu/127.10.238.252@KRBTEST.COM
I20260929 10:59:04.601507 1530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:04.594172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.253:46213 (local address 127.0.0.1:49416)
0929 10:59:04.594250 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0929 10:59:04.594259 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 10:59:04.594298 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:59:04.594415 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:59:04.594416 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:59:04.594522 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:59:04.594563 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:59:04.594564 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:59:04.595431 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:59:04.595432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:59:04.596069 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:59:04.596071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:59:04.596535 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:59:04.596536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:59:04.596608 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:04.596992 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:59:04.596997 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:59:04.598438 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:59:04.600380 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:59:04.600381 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:59:04.600382 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:59:04.600581 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:59:04.600823 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:59:04.600824 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:59:04.600824 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:59:04.600889 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:59:04.601197 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:59:04.601198 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:59:04.601243 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 10:59:04.601332 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 10:59:04.601419 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260929 10:59:04.601540 1457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:04.594244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49416 (local address 127.10.238.253:46213)
0929 10:59:04.594294 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:04.594296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:04.594305 (+ 9us) server_negotiation.cc:407] Connection header received
0929 10:59:04.594336 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:04.594336 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:04.594346 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:04.594407 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:04.594659 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:04.595358 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:04.596167 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:04.596451 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:04.596534 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:04.598534 (+ 2000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:59:04.598535 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:59:04.598535 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:59:04.598542 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:59:04.600301 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:04.600661 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:04.600663 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:04.600664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:04.600738 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:04.600953 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:04.600954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:04.600954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:04.601099 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:59:04.601166 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 10:59:04.601372 (+ 206us) server_negotiation.cc:299] Negotiation successful
0929 10:59:04.601451 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260929 10:59:04.602198 1528 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.238.252" port: 34939 } has no permanent_uuid. Determining permanent_uuid...
I20260929 10:59:04.603428 1380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:04.594865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.252:34939 (local address 127.0.0.1:35054)
0929 10:59:04.594908 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0929 10:59:04.594918 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 10:59:04.594954 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:59:04.595155 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:59:04.595156 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:59:04.595332 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:59:04.595373 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:59:04.595375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:59:04.596311 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:59:04.596312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:59:04.597171 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:59:04.597174 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:59:04.597716 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:59:04.597718 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:59:04.597777 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:04.598154 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:59:04.598160 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:59:04.599700 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:59:04.602358 (+ 2658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:59:04.602359 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:59:04.602360 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:59:04.602575 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:59:04.602779 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:59:04.602780 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:59:04.602780 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:59:04.602839 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:59:04.603118 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:59:04.603119 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:59:04.603164 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 10:59:04.603250 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 10:59:04.603337 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260929 10:59:04.603650 1533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:04.594897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35054 (local address 127.10.238.252:34939)
0929 10:59:04.595035 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:04.595036 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:04.595042 (+ 6us) server_negotiation.cc:407] Connection header received
0929 10:59:04.595065 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:04.595065 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:04.595072 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:04.595137 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:04.595466 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:04.596198 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:04.597288 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:04.597626 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:04.597928 (+ 302us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:04.599783 (+ 1855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:59:04.599784 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:59:04.599784 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:59:04.599790 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:59:04.602267 (+ 2477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:04.602661 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:04.602662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:04.602662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:04.602721 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:04.602910 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:04.602911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:04.602911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:04.603037 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:59:04.603110 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 10:59:04.603290 (+ 180us) server_negotiation.cc:299] Negotiation successful
0929 10:59:04.603367 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":81,"threads_started":1}
Sep 29 10:59:04 dist-test-slave-8nrt krb5kdc[1297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.238.254@KRBTEST.COM
I20260929 10:59:04.605758 1378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b0dce2aa03b4ad49f7d70a2c33f1d2d: Bootstrap starting.
Sep 29 10:59:04 dist-test-slave-8nrt krb5kdc[1297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679544, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.252@KRBTEST.COM for kudu/127.10.238.252@KRBTEST.COM
I20260929 10:59:04.606448 1378 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b0dce2aa03b4ad49f7d70a2c33f1d2d: Neither blocks nor log segments found. Creating new log.
I20260929 10:59:04.606792 1378 log.cc:824] T 00000000000000000000000000000000 P 0b0dce2aa03b4ad49f7d70a2c33f1d2d: Log is configured to *not* fsync() on all Append() calls
I20260929 10:59:04.607892 1378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b0dce2aa03b4ad49f7d70a2c33f1d2d: No bootstrap required, opened a new log
I20260929 10:59:04.608646 1381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:04.599646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57636 (local address 127.10.238.254:35497)
0929 10:59:04.599794 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:04.599796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:04.599802 (+ 6us) server_negotiation.cc:407] Connection header received
0929 10:59:04.599820 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:04.599821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:04.599832 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:04.599883 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:04.600461 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:04.601086 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:04.602100 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:04.602411 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:04.602470 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:04.605069 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:59:04.605070 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:59:04.605070 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:59:04.605077 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:59:04.606757 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:04.607198 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:04.607199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:04.607199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:04.607274 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:04.607496 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:04.607497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:04.607498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:04.607635 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:59:04.607790 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 10:59:04.608475 (+ 685us) server_negotiation.cc:299] Negotiation successful
0929 10:59:04.608546 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260929 10:59:04.609618 1532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:04.602480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35064 (local address 127.10.238.252:34939)
0929 10:59:04.602509 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:04.602511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:04.602533 (+ 22us) server_negotiation.cc:407] Connection header received
0929 10:59:04.602598 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:04.602599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:04.602608 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:04.602650 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:04.602938 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:04.603617 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:04.604464 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:04.604713 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:04.604752 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:04.606763 (+ 2011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:59:04.606764 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:59:04.606764 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:59:04.606771 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:59:04.608250 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:04.608641 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:04.608642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:04.608643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:04.608702 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:04.608980 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:04.608981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:04.608982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:04.609156 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:59:04.609216 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 10:59:04.609431 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 10:59:04.609510 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 10:59:04.609884 1530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:04.602454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.252:34939 (local address 127.0.0.1:35064)
0929 10:59:04.602493 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0929 10:59:04.602503 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 10:59:04.602546 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:59:04.602661 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:59:04.602662 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:59:04.602819 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:59:04.602856 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:59:04.602858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:59:04.603708 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:59:04.603709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:59:04.604374 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:59:04.604377 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:59:04.604817 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:59:04.604818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:59:04.604860 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:04.605285 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:59:04.605290 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:59:04.606656 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:59:04.608332 (+ 1676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:59:04.608332 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:59:04.608333 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:59:04.608558 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:59:04.608872 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:59:04.608872 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:59:04.608873 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:59:04.608928 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:59:04.609237 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:59:04.609238 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:59:04.609328 (+ 90us) client_negotiation.cc:769] Sending connection context
0929 10:59:04.609701 (+ 373us) client_negotiation.cc:240] Negotiation successful
0929 10:59:04.609792 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260929 10:59:04.610129 1378 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b0dce2aa03b4ad49f7d70a2c33f1d2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b0dce2aa03b4ad49f7d70a2c33f1d2d" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 35497 } } peers { permanent_uuid: "f05dbc3103cb4f9eb3519b44907b66a6" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 46213 } } peers { permanent_uuid: "732d35126b1f471cb64cce86634a9a11" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 34939 } }
I20260929 10:59:04.610265 1378 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b0dce2aa03b4ad49f7d70a2c33f1d2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b0dce2aa03b4ad49f7d70a2c33f1d2d, State: Initialized, Role: FOLLOWER
I20260929 10:59:04.610337 1378 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b0dce2aa03b4ad49f7d70a2c33f1d2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b0dce2aa03b4ad49f7d70a2c33f1d2d" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 35497 } } peers { permanent_uuid: "f05dbc3103cb4f9eb3519b44907b66a6" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 46213 } } peers { permanent_uuid: "732d35126b1f471cb64cce86634a9a11" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 34939 } }
I20260929 10:59:04.610597 1535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b0dce2aa03b4ad49f7d70a2c33f1d2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b0dce2aa03b4ad49f7d70a2c33f1d2d" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 35497 } } peers { permanent_uuid: "f05dbc3103cb4f9eb3519b44907b66a6" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 46213 } } peers { permanent_uuid: "732d35126b1f471cb64cce86634a9a11" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 34939 } } }
I20260929 10:59:04.610648 1378 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b0dce2aa03b4ad49f7d70a2c33f1d2d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:59:04.610683 1535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b0dce2aa03b4ad49f7d70a2c33f1d2d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:59:04.612247 1528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 732d35126b1f471cb64cce86634a9a11: Bootstrap starting.
I20260929 10:59:04.612882 1528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 732d35126b1f471cb64cce86634a9a11: Neither blocks nor log segments found. Creating new log.
I20260929 10:59:04.613085 1528 log.cc:824] T 00000000000000000000000000000000 P 732d35126b1f471cb64cce86634a9a11: Log is configured to *not* fsync() on all Append() calls
Sep 29 10:59:04 dist-test-slave-8nrt krb5kdc[1297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.238.253@KRBTEST.COM
I20260929 10:59:04.613987 1528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 732d35126b1f471cb64cce86634a9a11: No bootstrap required, opened a new log
W20260929 10:59:04.613798 1546 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0b0dce2aa03b4ad49f7d70a2c33f1d2d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:59:04.615089 1546 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:59:04.616412 1528 raft_consensus.cc:348] T 00000000000000000000000000000000 P 732d35126b1f471cb64cce86634a9a11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b0dce2aa03b4ad49f7d70a2c33f1d2d" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 35497 } } peers { permanent_uuid: "f05dbc3103cb4f9eb3519b44907b66a6" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 46213 } } peers { permanent_uuid: "732d35126b1f471cb64cce86634a9a11" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 34939 } }
I20260929 10:59:04.616642 1528 raft_consensus.cc:729] T 00000000000000000000000000000000 P 732d35126b1f471cb64cce86634a9a11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732d35126b1f471cb64cce86634a9a11, State: Initialized, Role: FOLLOWER
I20260929 10:59:04.616787 1528 consensus_queue.cc:260] T 00000000000000000000000000000000 P 732d35126b1f471cb64cce86634a9a11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b0dce2aa03b4ad49f7d70a2c33f1d2d" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 35497 } } peers { permanent_uuid: "f05dbc3103cb4f9eb3519b44907b66a6" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 46213 } } peers { permanent_uuid: "732d35126b1f471cb64cce86634a9a11" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 34939 } }
I20260929 10:59:04.616911 1457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:04.609150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49418 (local address 127.10.238.253:46213)
0929 10:59:04.609187 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:04.609188 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:04.609195 (+ 7us) server_negotiation.cc:407] Connection header received
0929 10:59:04.609216 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:04.609217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:04.609225 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:04.609349 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:04.609942 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:04.610612 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:04.611425 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:04.611630 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:04.611710 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:04.614276 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:59:04.614277 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:59:04.614277 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:59:04.614284 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:59:04.615729 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:04.616139 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:04.616141 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:04.616141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:04.616196 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:04.616389 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:04.616390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:04.616390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:04.616502 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:59:04.616653 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 10:59:04.616699 (+ 46us) server_negotiation.cc:299] Negotiation successful
0929 10:59:04.616781 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 10:59:04.617460 1528 sys_catalog.cc:564] T 00000000000000000000000000000000 P 732d35126b1f471cb64cce86634a9a11 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:59:04.614847 1310 master_runner.cc:186] Error getting master registration for 127.10.238.253:46213: OK, instance_id { permanent_uuid: "f05dbc3103cb4f9eb3519b44907b66a6" instance_seqno: 1790679544426731 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 10:59:04.618171 1310 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 10:59:04.618218 11195 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:59:04.619057 1352 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "0b0dce2aa03b4ad49f7d70a2c33f1d2d"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57636
I20260929 10:59:04.619201 1352 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b0dce2aa03b4ad49f7d70a2c33f1d2d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 10:59:04.619251 1352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b0dce2aa03b4ad49f7d70a2c33f1d2d [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:59:04.620010 1352 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b0dce2aa03b4ad49f7d70a2c33f1d2d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b0dce2aa03b4ad49f7d70a2c33f1d2d" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 35497 } } peers { permanent_uuid: "f05dbc3103cb4f9eb3519b44907b66a6" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 46213 } } peers { permanent_uuid: "732d35126b1f471cb64cce86634a9a11" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 34939 } }
I20260929 10:59:04.620273 1352 leader_election.cc:288] T 00000000000000000000000000000000 P 0b0dce2aa03b4ad49f7d70a2c33f1d2d [CANDIDATE]: Term 1 election: Requested vote from peers f05dbc3103cb4f9eb3519b44907b66a6 (127.10.238.253:46213), 732d35126b1f471cb64cce86634a9a11 (127.10.238.252:34939)
I20260929 10:59:04.621016 1549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 732d35126b1f471cb64cce86634a9a11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b0dce2aa03b4ad49f7d70a2c33f1d2d" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 35497 } } peers { permanent_uuid: "f05dbc3103cb4f9eb3519b44907b66a6" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 46213 } } peers { permanent_uuid: "732d35126b1f471cb64cce86634a9a11" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 34939 } } }
I20260929 10:59:04.621162 1549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 732d35126b1f471cb64cce86634a9a11 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 10:59:04.622257 1560 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 732d35126b1f471cb64cce86634a9a11: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:59:04.622318 1560 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:59:04.622722 1502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0b0dce2aa03b4ad49f7d70a2c33f1d2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "732d35126b1f471cb64cce86634a9a11"
I20260929 10:59:04.622826 1502 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 732d35126b1f471cb64cce86634a9a11 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:59:04.623708 1502 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 732d35126b1f471cb64cce86634a9a11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b0dce2aa03b4ad49f7d70a2c33f1d2d in term 1.
I20260929 10:59:04.623946 1327 leader_election.cc:302] T 00000000000000000000000000000000 P 0b0dce2aa03b4ad49f7d70a2c33f1d2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b0dce2aa03b4ad49f7d70a2c33f1d2d, 732d35126b1f471cb64cce86634a9a11; no voters:
I20260929 10:59:04.624289 1535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b0dce2aa03b4ad49f7d70a2c33f1d2d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:59:04.624464 1535 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b0dce2aa03b4ad49f7d70a2c33f1d2d [term 1 LEADER]: Becoming Leader. State: Replica: 0b0dce2aa03b4ad49f7d70a2c33f1d2d, State: Running, Role: LEADER
I20260929 10:59:04.624540 1535 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b0dce2aa03b4ad49f7d70a2c33f1d2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b0dce2aa03b4ad49f7d70a2c33f1d2d" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 35497 } } peers { permanent_uuid: "f05dbc3103cb4f9eb3519b44907b66a6" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 46213 } } peers { permanent_uuid: "732d35126b1f471cb64cce86634a9a11" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 34939 } }
I20260929 10:59:04.625000 1535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b0dce2aa03b4ad49f7d70a2c33f1d2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b0dce2aa03b4ad49f7d70a2c33f1d2d. Latest consensus state: current_term: 1 leader_uuid: "0b0dce2aa03b4ad49f7d70a2c33f1d2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b0dce2aa03b4ad49f7d70a2c33f1d2d" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 35497 } } peers { permanent_uuid: "f05dbc3103cb4f9eb3519b44907b66a6" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 46213 } } peers { permanent_uuid: "732d35126b1f471cb64cce86634a9a11" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 34939 } } }
I20260929 10:59:04.625089 1535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b0dce2aa03b4ad49f7d70a2c33f1d2d [sys.catalog]: This master's current role is: LEADER
I20260929 10:59:04.626355 1460 master_runner.cc:186] Error getting master registration for 127.10.238.253:46213: OK, instance_id { permanent_uuid: "f05dbc3103cb4f9eb3519b44907b66a6" instance_seqno: 1790679544426731 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 10:59:04.628635 1460 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 10:59:04.628948 1571 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:59:04.629439 1532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:04.621386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35080 (local address 127.10.238.252:34939)
0929 10:59:04.621431 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:04.621432 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:04.621551 (+ 119us) server_negotiation.cc:407] Connection header received
0929 10:59:04.621665 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:04.621666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:04.621675 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:04.621740 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:04.622496 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:04.623233 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:04.624332 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:04.624606 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:04.624652 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:04.626392 (+ 1740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:59:04.626393 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:59:04.626394 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:59:04.626401 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:59:04.628051 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:04.628442 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:04.628443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:04.628443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:04.628497 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:04.628794 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:04.628795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:04.628796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:04.628933 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:59:04.629004 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 10:59:04.629247 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 10:59:04.629353 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 10:59:04.630748 1571 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:59:04.635850 1381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:04.625304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57652 (local address 127.10.238.254:35497)
0929 10:59:04.625341 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:04.625343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:04.625353 (+ 10us) server_negotiation.cc:407] Connection header received
0929 10:59:04.625375 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:04.625376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:04.625386 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:04.625441 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:04.628883 (+ 3442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:04.629621 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:04.630606 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:04.630914 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:04.630993 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:04.632833 (+ 1840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:59:04.632834 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:59:04.632835 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:59:04.632842 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:59:04.634445 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:04.635039 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:04.635040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:04.635040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:04.635096 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:04.635286 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:04.635287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:04.635287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:04.635403 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:59:04.635486 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 10:59:04.635674 (+ 188us) server_negotiation.cc:299] Negotiation successful
0929 10:59:04.635761 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 10:59:04.639340 1457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:04.624341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49424 (local address 127.10.238.253:46213)
0929 10:59:04.624378 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:04.624380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:04.629905 (+ 5525us) server_negotiation.cc:407] Connection header received
0929 10:59:04.630111 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:04.630112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:04.630122 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:04.630149 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:04.630898 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:04.631824 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:04.632780 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:04.634188 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:04.634234 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:04.636291 (+ 2057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:59:04.636293 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:59:04.636293 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:59:04.636300 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:59:04.638119 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:04.638510 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:04.638511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:04.638511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:04.638568 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:04.638772 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:04.638773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:04.638773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:04.638899 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:59:04.638971 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 10:59:04.639171 (+ 200us) server_negotiation.cc:299] Negotiation successful
0929 10:59:04.639251 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 10:59:04.640374 1502 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 732d35126b1f471cb64cce86634a9a11 [term 1 FOLLOWER]: Refusing update from remote peer 0b0dce2aa03b4ad49f7d70a2c33f1d2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:59:04.640707 1568 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0b0dce2aa03b4ad49f7d70a2c33f1d2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "732d35126b1f471cb64cce86634a9a11" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 34939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:59:04.642144 1549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 732d35126b1f471cb64cce86634a9a11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b0dce2aa03b4ad49f7d70a2c33f1d2d. Latest consensus state: current_term: 1 leader_uuid: "0b0dce2aa03b4ad49f7d70a2c33f1d2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b0dce2aa03b4ad49f7d70a2c33f1d2d" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 35497 } } peers { permanent_uuid: "f05dbc3103cb4f9eb3519b44907b66a6" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 46213 } } peers { permanent_uuid: "732d35126b1f471cb64cce86634a9a11" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 34939 } } }
I20260929 10:59:04.642231 1549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 732d35126b1f471cb64cce86634a9a11 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:59:04.643694 1535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b0dce2aa03b4ad49f7d70a2c33f1d2d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0b0dce2aa03b4ad49f7d70a2c33f1d2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b0dce2aa03b4ad49f7d70a2c33f1d2d" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 35497 } } peers { permanent_uuid: "f05dbc3103cb4f9eb3519b44907b66a6" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 46213 } } peers { permanent_uuid: "732d35126b1f471cb64cce86634a9a11" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 34939 } } }
I20260929 10:59:04.643782 1535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b0dce2aa03b4ad49f7d70a2c33f1d2d [sys.catalog]: This master's current role is: LEADER
I20260929 10:59:04.644193 1549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 732d35126b1f471cb64cce86634a9a11 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0b0dce2aa03b4ad49f7d70a2c33f1d2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b0dce2aa03b4ad49f7d70a2c33f1d2d" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 35497 } } peers { permanent_uuid: "f05dbc3103cb4f9eb3519b44907b66a6" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 46213 } } peers { permanent_uuid: "732d35126b1f471cb64cce86634a9a11" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 34939 } } }
I20260929 10:59:04.644268 1549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 732d35126b1f471cb64cce86634a9a11 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:59:04.644640 1571 catalog_manager.cc:1347] Generated new cluster ID: 6bca637b08f4436181173babd6850bae
I20260929 10:59:04.644685 1571 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:59:04.645675 1573 mvcc.cc:205] Tried to move back new op lower bound from 7334623414844674048 to 7334623414782894080. Current Snapshot: MvccSnapshot[applied={T|T < 7334623414844674048}]
I20260929 10:59:04.654179 1571 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:59:04.655012 1571 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:59:04.660569 1453 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.238.252" port: 34939 } attempt: 3
I20260929 10:59:04.662667 1571 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b0dce2aa03b4ad49f7d70a2c33f1d2d: Generated new TSK 0
I20260929 10:59:04.662985 1571 catalog_manager.cc:1514] Initializing in-progress tserver states...
Sep 29 10:59:04 dist-test-slave-8nrt krb5kdc[1297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790679544, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.238.253@KRBTEST.COM for kudu/127.10.238.252@KRBTEST.COM
I20260929 10:59:04.670224 1455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:04.660866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.238.252:34939 (local address 127.0.0.1:35092)
0929 10:59:04.661048 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 10:59:04.661060 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:59:04.661104 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:59:04.661439 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:59:04.661440 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:59:04.661628 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:59:04.661677 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:59:04.661679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:59:04.662584 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:59:04.662586 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:59:04.663393 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:59:04.663396 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:59:04.663855 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:59:04.663855 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:59:04.663912 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:04.664438 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:59:04.664444 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:59:04.665967 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:59:04.667891 (+ 1924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:59:04.667892 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:59:04.667893 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:59:04.668125 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:59:04.669120 (+ 995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:59:04.669121 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:59:04.669121 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:59:04.669211 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:59:04.669897 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:59:04.669898 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:59:04.669945 (+ 47us) client_negotiation.cc:769] Sending connection context
0929 10:59:04.670017 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 10:59:04.670121 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159}
I20260929 10:59:04.670276 1532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:04.661236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35092 (local address 127.10.238.252:34939)
0929 10:59:04.661298 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:04.661300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:04.661307 (+ 7us) server_negotiation.cc:407] Connection header received
0929 10:59:04.661327 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:04.661328 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:04.661336 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:04.661390 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:04.661762 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:04.662495 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:04.663471 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:04.663782 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:04.663854 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:04.666060 (+ 2206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:59:04.666062 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:59:04.666062 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:59:04.666070 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:59:04.667769 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:04.668975 (+ 1206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:04.668976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:04.668976 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:04.669041 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:59:04.669452 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:59:04.669453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:59:04.669454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:59:04.669615 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:59:04.669876 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0929 10:59:04.670099 (+ 223us) server_negotiation.cc:299] Negotiation successful
0929 10:59:04.670184 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 10:59:04.671890 1453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f05dbc3103cb4f9eb3519b44907b66a6: Bootstrap starting.
I20260929 10:59:04.672629 1453 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f05dbc3103cb4f9eb3519b44907b66a6: Neither blocks nor log segments found. Creating new log.
I20260929 10:59:04.672933 1453 log.cc:824] T 00000000000000000000000000000000 P f05dbc3103cb4f9eb3519b44907b66a6: Log is configured to *not* fsync() on all Append() calls
I20260929 10:59:04.677582 1453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f05dbc3103cb4f9eb3519b44907b66a6: No bootstrap required, opened a new log
I20260929 10:59:04.679720 1453 raft_consensus.cc:348] T 00000000000000000000000000000000 P f05dbc3103cb4f9eb3519b44907b66a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b0dce2aa03b4ad49f7d70a2c33f1d2d" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 35497 } } peers { permanent_uuid: "f05dbc3103cb4f9eb3519b44907b66a6" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 46213 } } peers { permanent_uuid: "732d35126b1f471cb64cce86634a9a11" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 34939 } }
I20260929 10:59:04.679867 1453 raft_consensus.cc:729] T 00000000000000000000000000000000 P f05dbc3103cb4f9eb3519b44907b66a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f05dbc3103cb4f9eb3519b44907b66a6, State: Initialized, Role: FOLLOWER
I20260929 10:59:04.679951 1453 consensus_queue.cc:260] T 00000000000000000000000000000000 P f05dbc3103cb4f9eb3519b44907b66a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b0dce2aa03b4ad49f7d70a2c33f1d2d" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 35497 } } peers { permanent_uuid: "f05dbc3103cb4f9eb3519b44907b66a6" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 46213 } } peers { permanent_uuid: "732d35126b1f471cb64cce86634a9a11" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 34939 } }
I20260929 10:59:04.680260 1579 sys_catalog.cc:455] T 00000000000000000000000000000000 P f05dbc3103cb4f9eb3519b44907b66a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b0dce2aa03b4ad49f7d70a2c33f1d2d" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 35497 } } peers { permanent_uuid: "f05dbc3103cb4f9eb3519b44907b66a6" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 46213 } } peers { permanent_uuid: "732d35126b1f471cb64cce86634a9a11" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 34939 } } }
I20260929 10:59:04.680315 1427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0b0dce2aa03b4ad49f7d70a2c33f1d2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f05dbc3103cb4f9eb3519b44907b66a6"
I20260929 10:59:04.680536 1579 sys_catalog.cc:458] T 00000000000000000000000000000000 P f05dbc3103cb4f9eb3519b44907b66a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:59:04.680380 1426 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f05dbc3103cb4f9eb3519b44907b66a6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:59:04.681624 1426 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f05dbc3103cb4f9eb3519b44907b66a6 [term 1 FOLLOWER]: Refusing update from remote peer 0b0dce2aa03b4ad49f7d70a2c33f1d2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:59:04.681998 1568 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0b0dce2aa03b4ad49f7d70a2c33f1d2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f05dbc3103cb4f9eb3519b44907b66a6" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 46213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:59:04.682499 1427 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f05dbc3103cb4f9eb3519b44907b66a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b0dce2aa03b4ad49f7d70a2c33f1d2d in term 1.
I20260929 10:59:04.680272 1453 sys_catalog.cc:564] T 00000000000000000000000000000000 P f05dbc3103cb4f9eb3519b44907b66a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:59:04.683940 1579 sys_catalog.cc:455] T 00000000000000000000000000000000 P f05dbc3103cb4f9eb3519b44907b66a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b0dce2aa03b4ad49f7d70a2c33f1d2d. Latest consensus state: current_term: 1 leader_uuid: "0b0dce2aa03b4ad49f7d70a2c33f1d2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b0dce2aa03b4ad49f7d70a2c33f1d2d" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 35497 } } peers { permanent_uuid: "f05dbc3103cb4f9eb3519b44907b66a6" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 46213 } } peers { permanent_uuid: "732d35126b1f471cb64cce86634a9a11" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 34939 } } }
I20260929 10:59:04.684092 1579 sys_catalog.cc:458] T 00000000000000000000000000000000 P f05dbc3103cb4f9eb3519b44907b66a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:59:04.684168 1579 sys_catalog.cc:455] T 00000000000000000000000000000000 P f05dbc3103cb4f9eb3519b44907b66a6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0b0dce2aa03b4ad49f7d70a2c33f1d2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b0dce2aa03b4ad49f7d70a2c33f1d2d" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 35497 } } peers { permanent_uuid: "f05dbc3103cb4f9eb3519b44907b66a6" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 46213 } } peers { permanent_uuid: "732d35126b1f471cb64cce86634a9a11" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 34939 } } }
I20260929 10:59:04.684469 1579 sys_catalog.cc:458] T 00000000000000000000000000000000 P f05dbc3103cb4f9eb3519b44907b66a6 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 10:59:04.684839 1593 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f05dbc3103cb4f9eb3519b44907b66a6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:59:04.684899 1593 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:59:04.693897 1585 mvcc.cc:205] Tried to move back new op lower bound from 7334623414932525056 to 7334623414782894080. Current Snapshot: MvccSnapshot[applied={T|T < 7334623414844674048}]
W20260929 10:59:04.706777 1457 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:59:04.704868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49426 (local address 127.10.238.253:46213)
0929 10:59:04.704913 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:04.704915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:04.706060 (+ 1145us) server_negotiation.cc:407] Connection header received
0929 10:59:04.706170 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:04.706171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:04.706182 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:04.706213 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:04.706695 (+ 482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49426: BlockingRecv error: recv got EOF from 127.0.0.1:49426 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260929 10:59:04.707551 1381 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:59:04.705326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57656 (local address 127.10.238.254:35497)
0929 10:59:04.705365 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:04.705367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:04.707033 (+ 1666us) server_negotiation.cc:407] Connection header received
0929 10:59:04.707134 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:04.707135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:04.707146 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:04.707176 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:04.707490 (+ 314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57656: BlockingRecv error: recv got EOF from 127.0.0.1:57656 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260929 10:59:04.708305 1532 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:59:04.705716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35104 (local address 127.10.238.252:34939)
0929 10:59:04.705758 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:04.705760 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:04.707788 (+ 2028us) server_negotiation.cc:407] Connection header received
0929 10:59:04.707888 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:04.707890 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:04.707900 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:04.707931 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:04.708243 (+ 312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35104: BlockingRecv error: recv got EOF from 127.0.0.1:35104 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260929 10:59:04.710776 1457 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:59:04.709918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49438 (local address 127.10.238.253:46213)
0929 10:59:04.709952 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:04.709954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:04.710223 (+ 269us) server_negotiation.cc:407] Connection header received
0929 10:59:04.710332 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:04.710333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:04.710350 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:04.710381 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:04.710735 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49438: BlockingRecv error: recv got EOF from 127.0.0.1:49438 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260929 10:59:04.713798 1457 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:59:04.712999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49448 (local address 127.10.238.253:46213)
0929 10:59:04.713030 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:04.713032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:04.713311 (+ 279us) server_negotiation.cc:407] Connection header received
0929 10:59:04.713406 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:04.713407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:04.713416 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:04.713443 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:04.713758 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49448: BlockingRecv error: recv got EOF from 127.0.0.1:49448 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260929 10:59:04.717602 1457 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:59:04.716794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49458 (local address 127.10.238.253:46213)
0929 10:59:04.716833 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:04.716835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:04.717065 (+ 230us) server_negotiation.cc:407] Connection header received
0929 10:59:04.717159 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:04.717160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:04.717170 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:04.717200 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:04.717564 (+ 364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49458: BlockingRecv error: recv got EOF from 127.0.0.1:49458 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260929 10:59:04.723469 1457 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:59:04.722743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49460 (local address 127.10.238.253:46213)
0929 10:59:04.722776 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:04.722778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:04.722999 (+ 221us) server_negotiation.cc:407] Connection header received
0929 10:59:04.723093 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:04.723094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:04.723104 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:04.723131 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:04.723432 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49460: BlockingRecv error: recv got EOF from 127.0.0.1:49460 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260929 10:59:04.733418 1457 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:59:04.732558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49470 (local address 127.10.238.253:46213)
0929 10:59:04.732593 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:04.732595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:04.732863 (+ 268us) server_negotiation.cc:407] Connection header received
0929 10:59:04.732962 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:04.732963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:04.732974 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:04.733003 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:04.733372 (+ 369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49470: BlockingRecv error: recv got EOF from 127.0.0.1:49470 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260929 10:59:04.751771 1457 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:59:04.750714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49476 (local address 127.10.238.253:46213)
0929 10:59:04.750757 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:04.750759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:04.751073 (+ 314us) server_negotiation.cc:407] Connection header received
0929 10:59:04.751171 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:04.751172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:04.751184 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:04.751216 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:04.751704 (+ 488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49476: BlockingRecv error: recv got EOF from 127.0.0.1:49476 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260929 10:59:04.785954 1457 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:59:04.785123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49488 (local address 127.10.238.253:46213)
0929 10:59:04.785172 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:04.785174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:04.785329 (+ 155us) server_negotiation.cc:407] Connection header received
0929 10:59:04.785401 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:04.785402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:04.785413 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:04.785483 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:04.785891 (+ 408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49488: BlockingRecv error: recv got EOF from 127.0.0.1:49488 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260929 10:59:04.851874 1457 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:59:04.851139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49498 (local address 127.10.238.253:46213)
0929 10:59:04.851196 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:04.851198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:04.851276 (+ 78us) server_negotiation.cc:407] Connection header received
0929 10:59:04.851343 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:04.851344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:04.851357 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:04.851419 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:04.851811 (+ 392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49498: BlockingRecv error: recv got EOF from 127.0.0.1:49498 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260929 10:59:04.981680 1457 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:59:04.980877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49514 (local address 127.10.238.253:46213)
0929 10:59:04.980963 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:04.980966 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:04.981059 (+ 93us) server_negotiation.cc:407] Connection header received
0929 10:59:04.981156 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:04.981157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:04.981169 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:04.981238 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:04.981602 (+ 364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49514: BlockingRecv error: recv got EOF from 127.0.0.1:49514 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 10:59:05.231789 1568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b0dce2aa03b4ad49f7d70a2c33f1d2d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0b0dce2aa03b4ad49f7d70a2c33f1d2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b0dce2aa03b4ad49f7d70a2c33f1d2d" member_type: VOTER last_known_addr { host: "127.10.238.254" port: 35497 } } peers { permanent_uuid: "f05dbc3103cb4f9eb3519b44907b66a6" member_type: VOTER last_known_addr { host: "127.10.238.253" port: 46213 } } peers { permanent_uuid: "732d35126b1f471cb64cce86634a9a11" member_type: VOTER last_known_addr { host: "127.10.238.252" port: 34939 } } }
I20260929 10:59:05.231910 1568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b0dce2aa03b4ad49f7d70a2c33f1d2d [sys.catalog]: This master's current role is: LEADER
W20260929 10:59:05.239409 1457 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:59:05.238785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49524 (local address 127.10.238.253:46213)
0929 10:59:05.238865 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:05.238867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:05.238923 (+ 56us) server_negotiation.cc:407] Connection header received
0929 10:59:05.238980 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:05.238981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:05.238991 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:05.239049 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:59:05.239364 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49524: BlockingRecv error: recv got EOF from 127.0.0.1:49524 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 10:59:05.622759 1560 catalog_manager.cc:1259] Loaded cluster ID: 6bca637b08f4436181173babd6850bae
I20260929 10:59:05.622851 1560 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 732d35126b1f471cb64cce86634a9a11: loading cluster ID for follower catalog manager: success
I20260929 10:59:05.623636 1560 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 732d35126b1f471cb64cce86634a9a11: acquiring CA information for follower catalog manager: success
I20260929 10:59:05.623878 1560 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 732d35126b1f471cb64cce86634a9a11: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 10:59:05.685415 1593 catalog_manager.cc:1259] Loaded cluster ID: 6bca637b08f4436181173babd6850bae
I20260929 10:59:05.685491 1593 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f05dbc3103cb4f9eb3519b44907b66a6: loading cluster ID for follower catalog manager: success
I20260929 10:59:05.686396 1593 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f05dbc3103cb4f9eb3519b44907b66a6: acquiring CA information for follower catalog manager: success
I20260929 10:59:05.686745 1593 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f05dbc3103cb4f9eb3519b44907b66a6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 10:59:05.755123 1663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:05.752567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33964 (local address 127.10.238.253:46213)
0929 10:59:05.752695 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:05.752697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:05.752745 (+ 48us) server_negotiation.cc:407] Connection header received
0929 10:59:05.752812 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:05.752813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:05.752899 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:05.752973 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:59:05.753106 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:05.753647 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:05.754418 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:05.754606 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:05.754654 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:05.754789 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:59:05.754897 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:59:05.754953 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 10:59:05.754990 (+ 37us) server_negotiation.cc:299] Negotiation successful
0929 10:59:05.755007 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":58,"threads_started":1}
I20260929 10:59:05.759337 1669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:05.757096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40738 (local address 127.10.238.252:34939)
0929 10:59:05.757188 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:05.757190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:05.757198 (+ 8us) server_negotiation.cc:407] Connection header received
0929 10:59:05.757303 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:05.757304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:05.757340 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:05.757412 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:59:05.757497 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:05.757981 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:05.758650 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:05.758863 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:05.758911 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:05.759027 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:59:05.759114 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:59:05.759174 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 10:59:05.759209 (+ 35us) server_negotiation.cc:299] Negotiation successful
0929 10:59:05.759225 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":46,"threads_started":1}
I20260929 10:59:05.763708 1669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:05.761175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40740 (local address 127.10.238.252:34939)
0929 10:59:05.761237 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:05.761239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:05.761248 (+ 9us) server_negotiation.cc:407] Connection header received
0929 10:59:05.761306 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:05.761307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:05.761316 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:05.761360 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:59:05.761502 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:05.762049 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:05.762979 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:05.763274 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:05.763344 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:05.763466 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:59:05.763495 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:59:05.763549 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 10:59:05.763617 (+ 68us) server_negotiation.cc:299] Negotiation successful
0929 10:59:05.763637 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 10:59:05.764012 1663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:05.760981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33970 (local address 127.10.238.253:46213)
0929 10:59:05.761019 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:05.761021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:05.761437 (+ 416us) server_negotiation.cc:407] Connection header received
0929 10:59:05.761489 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:05.761490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:05.761501 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:05.761549 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:59:05.761715 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:05.762415 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:05.763219 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:05.763516 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:05.763557 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:05.763748 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:59:05.763781 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:59:05.763833 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 10:59:05.763923 (+ 90us) server_negotiation.cc:299] Negotiation successful
0929 10:59:05.763945 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 10:59:05.764379 1676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:59:05.760990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56136 (local address 127.10.238.254:35497)
0929 10:59:05.761094 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0929 10:59:05.761096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 10:59:05.761107 (+ 11us) server_negotiation.cc:407] Connection header received
0929 10:59:05.761227 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:59:05.761228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:59:05.761302 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:59:05.761379 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:59:05.761553 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:05.762314 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:05.763229 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:59:05.763556 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:59:05.763619 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:59:05.764013 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:59:05.764117 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:59:05.764196 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 10:59:05.764266 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 10:59:05.764292 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":53,"threads_started":1}
I20260929 10:59:05.765857 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 1310
I20260929 10:59:05.771508 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 1385
I20260929 10:59:05.776868 11195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc60rDW/build/release/bin/kudu with pid 1460
2026-09-29T10:59:05Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2820 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5656 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (271274 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST